Aug 04 17:58:41.274709 ip-10-0-133-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 04 17:58:41.274720 ip-10-0-133-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 04 17:58:41.274728 ip-10-0-133-250 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 04 17:58:41.274975 ip-10-0-133-250 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 04 17:58:52.686253 ip-10-0-133-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 04 17:58:52.686267 ip-10-0-133-250 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f6b2f566add445cb8eb84daf1bf0a4ba --
Aug 04 18:00:28.405829 ip-10-0-133-250 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 18:00:28.806080 ip-10-0-133-250 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:28.806080 ip-10-0-133-250 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 18:00:28.806080 ip-10-0-133-250 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:28.806080 ip-10-0-133-250 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 18:00:28.806080 ip-10-0-133-250 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:28.807968 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.807878    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 18:00:28.811449 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811429    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.811449 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811448    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811452    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811456    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811459    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811463    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811466    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811469    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811472    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811476    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811480    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811484    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811487    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811490    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811506    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811510    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811513    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811517    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811520    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811522    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.811537 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811526    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811529    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811532    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811535    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811537    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811542    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811546    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811549    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811553    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811557    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811561    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811564    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811567    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811570    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811573    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811576    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811578    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811581    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811584    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811587    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.812026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811590    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811592    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811595    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811597    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811600    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811602    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811605    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811607    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811610    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811613    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811616    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811618    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811621    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811624    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811627    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811630    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811633    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811636    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811638    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811641    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.812527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811643    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811646    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811649    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811651    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811654    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811657    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811659    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811662    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811665    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811667    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811670    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811673    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811675    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811678    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811681    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811683    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811686    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811688    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811691    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811693    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.813032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811697    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811700    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811702    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811705    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811707    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811710    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811714    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811716    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811719    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811722    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811725    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811727    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811730    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811732    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811735    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811737    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811740    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811742    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.811745    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.813539 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812543    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812551    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812554    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812557    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812560    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812563    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812566    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812569    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812572    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812574    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812577    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812580    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812583    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812585    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812588    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812591    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812593    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812596    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812598    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812602    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.814010 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812605    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812608    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812611    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812614    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812617    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812619    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812622    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812624    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812627    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812629    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812632    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812635    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812637    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812640    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812642    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812645    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812647    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812651    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812654    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812656    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.814527 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812659    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812662    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812665    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812667    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812670    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812673    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812675    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812678    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812680    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812683    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812685    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812688    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812690    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812693    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812696    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812698    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812700    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812704    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812706    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.815026 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812709    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812711    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812714    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812716    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812719    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812721    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812725    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812729    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812731    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812734    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812736    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812739    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812743    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812745    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812748    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812750    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812753    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812755    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812758    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812760    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.815540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812763    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812765    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812768    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812770    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812773    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812775    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812778    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812782    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812785    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812790    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812793    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812798    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812801    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812804    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812808    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812811    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812815    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812817    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812820    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.816034 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.812823    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813226    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813231    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813235    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813237    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813241    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813243    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813246    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813249    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813251    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813254    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813256    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813259    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813261    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813264    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813266    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813269    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813272    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813275    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813277    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.816516 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813280    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813282    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813286    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813289    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813291    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813294    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813297    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813300    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813304    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813307    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813311    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813314    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813317    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813320    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813323    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813326    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813329    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813332    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813336    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.817032 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813339    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813342    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813345    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813347    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813350    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813353    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813356    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813358    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813361    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813364    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813367    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813370    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813373    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813375    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813378    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813382    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813385    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813389    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813391    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.817508 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813394    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813398    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813401    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813404    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813407    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813410    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813412    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813416    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813418    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813421    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813424    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813427    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813429    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813433    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813436    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813438    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813442    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813444    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813447    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.817971 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813449    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813451    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813454    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813456    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813459    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813462    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813464    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813467    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813469    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813473    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813476    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813478    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813481    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813484    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813486    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813489    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813491    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813508    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813510    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813513    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.818466 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813516    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813518    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.813521    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813603    2579 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813613    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813623    2579 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813630    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813636    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813639    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813644    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813648    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813651    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813654    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813658    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813662    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813668    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813671    2579 flags.go:64] FLAG: --cgroup-root=""
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813674    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813677    2579 flags.go:64] FLAG: --client-ca-file=""
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813680    2579 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813683    2579 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813690    2579 flags.go:64] FLAG: --cluster-domain=""
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813693    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813696    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 18:00:28.818952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813699    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813703    2579 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813707    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813710    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813713    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813722    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813734    2579 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813737    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813740    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813743    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813746    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813751    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813754    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813757    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813760    2579 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813763    2579 flags.go:64] FLAG: --enable-server="true"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813766    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813771    2579 flags.go:64] FLAG: --event-burst="100"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813774    2579 flags.go:64] FLAG: --event-qps="50"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813777    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813780    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813784    2579 flags.go:64] FLAG: --eviction-hard=""
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813788    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813792    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813795    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 18:00:28.819568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813798    2579 flags.go:64] FLAG: --eviction-soft=""
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813801    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813804    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813807    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813812    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813815    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813818    2579 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813821    2579 flags.go:64] FLAG: --feature-gates=""
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813825    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813828    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813831    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813835    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813838    2579 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813841    2579 flags.go:64] FLAG: --help="false"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813844    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-250.ec2.internal"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813847    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813850    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813853    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813856    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813860    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813863    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813866    2579 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813869    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 18:00:28.820179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813872    2579 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813875    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813878    2579 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813881    2579 flags.go:64] FLAG: --kube-reserved=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813884    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813887    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813890    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813894    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813897    2579 flags.go:64] FLAG: --lock-file=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813900    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813903    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813906    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813911    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813915    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813918    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813921    2579 flags.go:64] FLAG: --logging-format="text"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813924    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813928    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813931    2579 flags.go:64] FLAG: --manifest-url=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813934    2579 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813938    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813941    2579 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813946    2579 flags.go:64] FLAG: --max-pods="110"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813949    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813952    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 18:00:28.820773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813955    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813958    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813961    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813964    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813967    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813975    2579 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813978    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813981    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813984    2579 flags.go:64] FLAG: --pod-cidr=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.813988    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814001    2579 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814004    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814007    2579 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814010    2579 flags.go:64] FLAG: --port="10250"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814013    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814017    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c39eefd7aaf6adf8"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814021    2579 flags.go:64] FLAG: --qos-reserved=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814024    2579 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814027    2579 flags.go:64] FLAG: --register-node="true"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814030    2579 flags.go:64] FLAG: --register-with-taints=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814039    2579 flags.go:64] FLAG: --registry-burst="10"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814042    2579 flags.go:64] FLAG: --registry-qps="5"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814045    2579 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814048    2579 flags.go:64] FLAG: --reserved-memory=""
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814052    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 18:00:28.821373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814054    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814058    2579 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814060    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814063    2579 flags.go:64] FLAG: --runonce="false"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814066    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814069    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814072    2579 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814075    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814078    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814081    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814084    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814088    2579 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814091    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814094    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814097    2579 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814100    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814103    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814106    2579 flags.go:64] FLAG: --system-cgroups=""
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814109    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814115    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814118    2579 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814120    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814126    2579 flags.go:64] FLAG: --tls-min-version=""
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814129    2579 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814132    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 18:00:28.822027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814135    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814138    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814141    2579 flags.go:64] FLAG: --v="2"
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814145    2579 flags.go:64] FLAG: --version="false"
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814149    2579 flags.go:64] FLAG: --vmodule=""
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814153    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.814156    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814273    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814276    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814280    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814283    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814286    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814289    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814291    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814294    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814297    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814299    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814302    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814304    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814307    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814310    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.822641 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814312    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814315    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814317    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814320    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814323    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814326    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814329    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814331    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814335    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814338    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814341    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814344    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814346    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814349    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814352    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814355    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814357    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814361    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814365    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814369    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.823154 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814371    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814374    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814377    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814380    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814383    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814385    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814388    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814390    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814393    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814396    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814399    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814402    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814404    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814407    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814410    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814412    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814415    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814418    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814421    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814424    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.823726 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814427    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814430    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814433    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814435    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814438    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814440    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814443    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814447    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814450    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814453    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814456    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814458    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814461    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814464    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814466    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814469    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814471    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814474    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814476    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.824231 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814479    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814482    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814485    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814488    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814491    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814506    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814508    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814511    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814514    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814516    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814519    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814523    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814525    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814530    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814532    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814535    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814538    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814541    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814543    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814546    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.824707 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814548    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814551    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814553    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814556    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814560    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.814563    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.815200    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.821770    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.821788    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.822693    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.822700    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822761    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822769    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.825283 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822772    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822776    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822779    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822782    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822785    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822788    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822791    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822793    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822796    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822798    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822801    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822804    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822807    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822809    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822812    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822816    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822820    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822822    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822825    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.825626 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822828    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822830    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822833    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822836    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822839    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822842    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822844    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822847    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822851    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822854    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822857    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822859    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822862    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822865    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822867    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822870    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822872    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822875    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822878    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822880    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.826088 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822883    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822885    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822888    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822891    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822893    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822896    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822899    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822901    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822904    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822906    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822909    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822911    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822914    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822917    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822919    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822922    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822925    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822927    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822930    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822933    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.826586 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822936    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822940    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822943    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822945    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822947    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822950    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822952    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822955    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822957    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822960    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822962    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822965    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822967    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822970    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822973    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822975    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822978    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822980    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822983    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822986    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.827091 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822988    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822990    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822993    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822995    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.822998    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823000    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823003    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823005    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823008    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823011    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823014    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823017    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823022    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823025    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823028    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823031    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823034    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.827654 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823037    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.823042    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823181    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823186    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823190    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823193    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823196    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823199    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823202    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823205    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823208    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823211    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823213    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823216    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823218    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:28.828067 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823221    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823225    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823229    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823232    2579 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823235    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823237    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823242    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823245    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823248    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823251    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823253    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823257    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823261    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823264    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823267    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823273    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823277    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823280    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823282    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:28.828448 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823284    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823287    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823289    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823292    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823295    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823297    2579 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823300    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823302    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823305    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823308    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823310    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823313    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823316    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823318    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823321    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823323    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823326    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823328    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823331    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823333    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:28.828930 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823336    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823338    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823341    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823344    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823346    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823349    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823351    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823354    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823356    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823359    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823362    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823365    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823367    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823370    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823372    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823375    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823378    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823380    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823383    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823385    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:28.829424 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823388    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823390    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823393    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823395    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823398    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823400    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823402    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823405    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823408    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823410    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823414    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823416    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823418    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823421    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823424    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823426    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823429    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823431    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823434    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823437    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:28.829935 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823440    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823442    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823445    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823448    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823453    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:28.823456    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.823460    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.823604    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.826513    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.828030    2579 server.go:1019] "Starting client certificate rotation"
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.828138    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:28.830428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.828170    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:28.830822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.830807    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:28.837360 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.837334    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:28.846012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.845983    2579 log.go:25] "Validated CRI v1 runtime API"
Aug 04 18:00:28.848109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.848080    2579 log.go:25] "Validated CRI v1 image API"
Aug 04 18:00:28.848977 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.848954    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 18:00:28.853137 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.853105    2579 fs.go:135] Filesystem UUIDs: map[116e5d5d-1f96-4bd1-a35e-ada8e7bbaf82:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c21baa5c-6437-49de-8005-976f66991f46:/dev/nvme0n1p4]
Aug 04 18:00:28.853215 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.853134    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 18:00:28.858117 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.857994    2579 manager.go:216] Machine: {Timestamp:2026-08-04 18:00:28.856962049 +0000 UTC m=+0.328249335 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100876 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27fe2918875216d481771b199fadb5 SystemUUID:ec27fe29-1887-5216-d481-771b199fadb5 BootID:f6b2f566-add4-45cb-8eb8-4daf1bf0a4ba Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:a4:6e:5a:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:a4:6e:5a:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:00:28:b4:41:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 18:00:28.858117 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.858104    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 18:00:28.858264 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.858216    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 18:00:28.859333 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.859306    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 18:00:28.859473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.859335    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 18:00:28.860088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.860077    2579 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 18:00:28.860138 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.860096    2579 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 18:00:28.860138 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.860105    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:28.860796 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.860784    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:28.861608 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.861597    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 18:00:28.863457 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.863440    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:28.865304 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865291    2579 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:28.865444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865433    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 18:00:28.865491 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865467    2579 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 18:00:28.865491 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865482    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 18:00:28.865589 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865508    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 18:00:28.865589 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865532    2579 kubelet.go:398] "Adding apiserver pod source"
Aug 04 18:00:28.865589 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865544    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 18:00:28.865912 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865900    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:28.865970 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.865918    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:28.868811 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.868795    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 18:00:28.870453 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.870440    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 18:00:28.870510 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.870470    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 18:00:28.871612 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871587    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 18:00:28.871612 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871610    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871618    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871626    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871640    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871648    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871656    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871661    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871667    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871673    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871695    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 18:00:28.871720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.871703    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 18:00:28.872469 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.872458    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 18:00:28.872469 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.872467    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 18:00:28.875903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.875882    2579 server.go:1268] "Started kubelet"
Aug 04 18:00:28.875988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.875959    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 18:00:28.876067 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.876047    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 18:00:28.876680 ip-10-0-133-250 systemd[1]: Started Kubernetes Kubelet.
Aug 04 18:00:28.879926 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.879870    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 18:00:28.880259 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.880240    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 18:00:28.881200 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.880615    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 18:00:28.881200 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.880665    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 18:00:28.881200 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.880833    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 18:00:28.882882 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.882859    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 18:00:28.883844 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.883829    2579 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 18:00:28.888009 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.887993    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 18:00:28.888086 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888012    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:28.888158 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.888124    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:28.888819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888645    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 18:00:28.888819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888656    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 18:00:28.888819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888669    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 18:00:28.889003 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888827    2579 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 18:00:28.889003 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.888836    2579 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 18:00:28.889254 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889238    2579 factory.go:55] Registering systemd factory
Aug 04 18:00:28.889254 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889256    2579 factory.go:223] Registration of the systemd container factory successfully
Aug 04 18:00:28.889448 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889435    2579 factory.go:153] Registering CRI-O factory
Aug 04 18:00:28.889448 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889447    2579 factory.go:223] Registration of the crio container factory successfully
Aug 04 18:00:28.889604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889488    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 18:00:28.889604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889524    2579 factory.go:103] Registering Raw factory
Aug 04 18:00:28.889604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889537    2579 manager.go:1196] Started watching for new ooms in manager
Aug 04 18:00:28.889946 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.888608    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-250.ec2.internal.18c8ac0e1d6d38b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-250.ec2.internal,UID:ip-10-0-133-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-250.ec2.internal,},FirstTimestamp:2026-08-04 18:00:28.875618481 +0000 UTC m=+0.346905775,LastTimestamp:2026-08-04 18:00:28.875618481 +0000 UTC m=+0.346905775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-250.ec2.internal,}"
Aug 04 18:00:28.889946 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.889929    2579 manager.go:349] Starting recovery of all containers
Aug 04 18:00:28.890970 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.890939    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 18:00:28.890970 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.890959    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 18:00:28.900193 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.900176    2579 manager.go:354] Recovery completed
Aug 04 18:00:28.904353 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.904340    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:28.906213 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906186    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:28.906281 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906225    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:28.906281 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906243    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:28.906800 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906785    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 18:00:28.906800 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906798    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 18:00:28.906908 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906791    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:28.906908 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.906813    2579 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:28.908318 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908294    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:28.908408 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908335    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:28.908408 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908352    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:28.908729 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908710    2579 policy_none.go:49] "None policy: Start"
Aug 04 18:00:28.908780 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908737    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 18:00:28.908780 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.908678    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-250.ec2.internal.18c8ac0e1f400b74  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-250.ec2.internal,UID:ip-10-0-133-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-250.ec2.internal,},FirstTimestamp:2026-08-04 18:00:28.906212212 +0000 UTC m=+0.377499476,LastTimestamp:2026-08-04 18:00:28.906212212 +0000 UTC m=+0.377499476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-250.ec2.internal,}"
Aug 04 18:00:28.908889 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.908753    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 18:00:28.910121 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.910106    2579 policy_none.go:47] "Start"
Aug 04 18:00:28.914117 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.914072    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tgvx2"
Aug 04 18:00:28.919414 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.919350    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-250.ec2.internal.18c8ac0e1f406f90  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-250.ec2.internal,UID:ip-10-0-133-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-250.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-250.ec2.internal,},FirstTimestamp:2026-08-04 18:00:28.90623784 +0000 UTC m=+0.377525105,LastTimestamp:2026-08-04 18:00:28.90623784 +0000 UTC m=+0.377525105,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-250.ec2.internal,}"
Aug 04 18:00:28.928506 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.928357    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tgvx2"
Aug 04 18:00:28.933255 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.933188    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-250.ec2.internal.18c8ac0e1f409196  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-250.ec2.internal,UID:ip-10-0-133-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-250.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-250.ec2.internal,},FirstTimestamp:2026-08-04 18:00:28.90624655 +0000 UTC m=+0.377533815,LastTimestamp:2026-08-04 18:00:28.90624655 +0000 UTC m=+0.377533815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-250.ec2.internal,}"
Aug 04 18:00:28.938313 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938298    2579 manager.go:337] "Starting Device Plugin manager"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.938330    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938344    2579 server.go:87] "Starting device plugin registration server"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938612    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938625    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938717    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938803    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:28.938813    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.939310    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 18:00:28.957153 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:28.939347    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.023184 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.023154    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 18:00:29.024368 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.024351    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 18:00:29.024469 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.024389    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 18:00:29.024469 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.024414    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 18:00:29.024469 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.024451    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 18:00:29.027606 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.027588    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:29.038859 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.038845    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:29.039682 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.039663    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:29.039775 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.039697    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:29.039775 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.039707    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:29.039775 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.039733    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.048525 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.048509    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.048592 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.048532    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-250.ec2.internal\": node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.076716 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.076636    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.124774 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.124730    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"]
Aug 04 18:00:29.124942 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.124839    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:29.126609 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.126590    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:29.126705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.126628    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:29.126705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.126642    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:29.128027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128014    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:29.128141 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.128181 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128158    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:29.128894 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128878    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:29.128954 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128907    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:29.128954 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128922    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:29.129018 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128922    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:29.129018 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.128995    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:29.129018 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.129009    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:29.130115 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.130097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.130208 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.130128    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:29.130804 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.130787    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:29.130885 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.130811    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:29.130885 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.130827    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:29.151442 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.151413    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-250.ec2.internal\" not found" node="ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.156104 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.156079    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-250.ec2.internal\" not found" node="ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.177136 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.177097    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.190283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.190253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.190375 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.190289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.190375 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.190312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d77a65267e4463f8e5a16dfb51a44592-config\") pod \"kube-apiserver-proxy-ip-10-0-133-250.ec2.internal\" (UID: \"d77a65267e4463f8e5a16dfb51a44592\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.277522 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.277480    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.290859 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.290927 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.290927 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d77a65267e4463f8e5a16dfb51a44592-config\") pod \"kube-apiserver-proxy-ip-10-0-133-250.ec2.internal\" (UID: \"d77a65267e4463f8e5a16dfb51a44592\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.290927 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.291022 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d77a65267e4463f8e5a16dfb51a44592-config\") pod \"kube-apiserver-proxy-ip-10-0-133-250.ec2.internal\" (UID: \"d77a65267e4463f8e5a16dfb51a44592\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.291022 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.290942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c8ab47d99a99d58a14551ab35cc2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal\" (UID: \"a17c8ab47d99a99d58a14551ab35cc2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.378261 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.378180    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.454778 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.454751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.459063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.458952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:29.478742 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.478719    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.579249 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.579195    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.680299 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.680262    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.780910 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.780860    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.826900 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.826871    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 18:00:29.827301 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.827042    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 18:00:29.881456 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:29.881415    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-250.ec2.internal\" not found"
Aug 04 18:00:29.896345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.896319    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:29.932261 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.932163    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:55:29 +0000 UTC" deadline="2028-01-26 21:03:10.408701653 +0000 UTC"
Aug 04 18:00:29.932261 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.932206    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12963h2m40.476501682s"
Aug 04 18:00:29.936239 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.936218    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:29.988962 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:29.988929    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:30.006666 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.006647    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:30.009108 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.009093    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal"
Aug 04 18:00:30.017539 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.017520    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:30.058415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.058395    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:30.813696 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:30.813660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17c8ab47d99a99d58a14551ab35cc2c.slice/crio-727dd57d88e13e8a499e155920d55359d6ac59009062f83650d3bf3655167f24 WatchSource:0}: Error finding container 727dd57d88e13e8a499e155920d55359d6ac59009062f83650d3bf3655167f24: Status 404 returned error can't find the container with id 727dd57d88e13e8a499e155920d55359d6ac59009062f83650d3bf3655167f24
Aug 04 18:00:30.818993 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.818978    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:00:30.868914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.868883    2579 apiserver.go:52] "Watching apiserver"
Aug 04 18:00:30.878000 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.877977    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 18:00:30.880166 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.880142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v","openshift-image-registry/node-ca-d8j6c","openshift-multus/multus-4nbss","openshift-network-operator/iptables-alerter-ktn7p","openshift-ovn-kubernetes/ovnkube-node-dwcm5","kube-system/global-pull-secret-syncer-m5ldx","kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lz22x","openshift-dns/node-resolver-t8vf5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-4ktf9","openshift-multus/network-metrics-daemon-jw5pt","openshift-network-diagnostics/network-check-target-tglfd","kube-system/konnectivity-agent-9fwpg"]
Aug 04 18:00:30.882398 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.882383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:30.882470 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:30.882455    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:30.883800 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.883786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:30.885027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.885011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.886373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.886355    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.886454 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.886368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 18:00:30.886493 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.886476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:30.886975 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.886955    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rnb46\""
Aug 04 18:00:30.887063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.887032    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.887531 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.887512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.888128 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nd8mb\""
Aug 04 18:00:30.888192 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888145    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 18:00:30.888317 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.888743 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888635    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.888743 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 18:00:30.888899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.888884    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:30.889401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.889381    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.889735 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.889719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5szz\""
Aug 04 18:00:30.889818 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.889779    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.889867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.889842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 18:00:30.889986 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.889971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:30.890069 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:30.890033    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:30.891559 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.891542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:30.891664 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.891643    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 18:00:30.891768 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.891684    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.892018 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.891987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.892080 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.892017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 18:00:30.892172 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.892110    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 18:00:30.892172 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.892153    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 18:00:30.892333 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.892187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mnkb8\""
Aug 04 18:00:30.893270 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.893255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:30.894451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.894436    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.894705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.894691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 18:00:30.894944 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.894929    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.895043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.895002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:30.896032 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.896016    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7v2s4\""
Aug 04 18:00:30.896269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.896251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:30.896599 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.896579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.896716 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.896651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.896716 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.896664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pwk2m\""
Aug 04 18:00:30.897688 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.897658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ttb85\""
Aug 04 18:00:30.897794 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.897696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 18:00:30.897863 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.897810    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 18:00:30.898844 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.898821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.900891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.900924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7mgmx\""
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:30.900962    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-slash\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-netns\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-ovn\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk44x\" (UniqueName: \"kubernetes.io/projected/5082e209-4074-4742-9da7-0ce1bb74779f-kube-api-access-kk44x\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-system-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9722643-4aff-48fd-8bef-8abe70c56b18-host-slash\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-k8s-cni-cncf-io\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw47r\" (UniqueName: \"kubernetes.io/projected/d9722643-4aff-48fd-8bef-8abe70c56b18-kube-api-access-tw47r\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-os-release\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-hostroot\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-node-log\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-config\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.901642 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-env-overrides\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovn-node-metrics-cert\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpv4\" (UniqueName: \"kubernetes.io/projected/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-kube-api-access-srpv4\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9722643-4aff-48fd-8bef-8abe70c56b18-iptables-alerter-script\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5082e209-4074-4742-9da7-0ce1bb74779f-serviceca\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cni-binary-copy\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901913    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-socket-dir-parent\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.901967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-multus-certs\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxmhw\""
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5082e209-4074-4742-9da7-0ce1bb74779f-host\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cnibin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-kubelet\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-conf-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.902444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-systemd-units\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-etc-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-log-socket\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-bin\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-kubelet\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-var-lib-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-netd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-dbus\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-bin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.902999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-etc-kubernetes\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-systemd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-script-lib\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:30.903504 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-kubelet-config\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:30.903504 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8nz\" (UniqueName: \"kubernetes.io/projected/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-kube-api-access-wn8nz\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903504 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-netns\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903504 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-multus\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903504 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-daemon-config\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:30.903739 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.903649    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:30.927393 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.927374    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6w2js"
Aug 04 18:00:30.936230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.936215    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6w2js"
Aug 04 18:00:30.979980 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:30.979961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77a65267e4463f8e5a16dfb51a44592.slice/crio-725eb178346b90643608ebdfed9121278c238128179a76ceb714269c917c8967 WatchSource:0}: Error finding container 725eb178346b90643608ebdfed9121278c238128179a76ceb714269c917c8967: Status 404 returned error can't find the container with id 725eb178346b90643608ebdfed9121278c238128179a76ceb714269c917c8967
Aug 04 18:00:30.989753 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:30.989733    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 18:00:31.003868 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw47r\" (UniqueName: \"kubernetes.io/projected/d9722643-4aff-48fd-8bef-8abe70c56b18-kube-api-access-tw47r\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.003961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkshf\" (UniqueName: \"kubernetes.io/projected/2550b4e7-088e-48e4-abc9-2a500311fabc-kube-api-access-qkshf\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.003961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysconfig\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.003961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-run\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-host\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.003987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-os-release\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-node-log\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-config\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-env-overrides\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-os-release\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-node-log\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovn-node-metrics-cert\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-socket-dir-parent\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21136c2d-8c4a-41d9-9b45-9d55c3599539-hosts-file\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-socket-dir-parent\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-conf-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-etc-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-log-socket\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-var-lib-kubelet\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-etc-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-env-overrides\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-log-socket\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-conf-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004631    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-config\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-system-cni-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-socket-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-var-lib-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.004761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-var-lib-openvswitch\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-bin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-etc-kubernetes\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-systemd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-bin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-kubelet-config\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-sys\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-os-release\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-kubelet-config\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-systemd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.004991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-etc-kubernetes\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8nz\" (UniqueName: \"kubernetes.io/projected/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-kube-api-access-wn8nz\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvt7\" (UniqueName: \"kubernetes.io/projected/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kube-api-access-hqvt7\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.005418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk44x\" (UniqueName: \"kubernetes.io/projected/5082e209-4074-4742-9da7-0ce1bb74779f-kube-api-access-kk44x\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-system-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9722643-4aff-48fd-8bef-8abe70c56b18-host-slash\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.005223    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-system-cni-dir\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gkj\" (UniqueName: \"kubernetes.io/projected/6642e299-8f13-49a5-835f-02c797ab231e-kube-api-access-z4gkj\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9722643-4aff-48fd-8bef-8abe70c56b18-host-slash\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-k8s-cni-cncf-io\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.005303    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.505280999 +0000 UTC m=+2.976568288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-k8s-cni-cncf-io\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-kubernetes\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-conf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-registration-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-device-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-sys-fs\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-hostroot\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.006164 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-hostroot\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpv4\" (UniqueName: \"kubernetes.io/projected/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-kube-api-access-srpv4\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9722643-4aff-48fd-8bef-8abe70c56b18-iptables-alerter-script\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5082e209-4074-4742-9da7-0ce1bb74779f-serviceca\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cni-binary-copy\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-multus-certs\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-multus-certs\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed50106d-71e8-4606-a3fd-081ac58e180c-konnectivity-ca\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-systemd\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5082e209-4074-4742-9da7-0ce1bb74779f-host\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-ovn-kubernetes\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cnibin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-kubelet\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5082e209-4074-4742-9da7-0ce1bb74779f-host\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-systemd-units\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007027 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-bin\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-dbus\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-kubelet\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-systemd-units\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.005963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-bin\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cnibin\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-etc-tuned\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-kubelet\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d06c764-6c99-4554-a19c-b8f08450baef-dbus\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5082e209-4074-4742-9da7-0ce1bb74779f-serviceca\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-netd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-kubelet\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed50106d-71e8-4606-a3fd-081ac58e180c-agent-certs\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-cnibin\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-cni-netd\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.007783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9722643-4aff-48fd-8bef-8abe70c56b18-iptables-alerter-script\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-cni-binary-copy\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-script-lib\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-lib-modules\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21136c2d-8c4a-41d9-9b45-9d55c3599539-tmp-dir\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrg2\" (UniqueName: \"kubernetes.io/projected/21136c2d-8c4a-41d9-9b45-9d55c3599539-kube-api-access-snrg2\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-modprobe-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-netns\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-multus\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-run-netns\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-daemon-config\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-slash\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-netns\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-host-var-lib-cni-multus\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-ovn\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-slash\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-etc-selinux\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-tmp\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-host-run-netns\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-run-ovn\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbdf\" (UniqueName: \"kubernetes.io/projected/126a405e-09cc-479a-acfa-68a3b51990ff-kube-api-access-mpbdf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovnkube-script-lib\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.006993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-multus-daemon-config\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.008841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.008600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-ovn-node-metrics-cert\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.012791 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.012775    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:31.012855 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.012795    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:31.012855 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.012807    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.012954 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.012859    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.51284589 +0000 UTC m=+2.984133143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.015345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.015317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpv4\" (UniqueName: \"kubernetes.io/projected/8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353-kube-api-access-srpv4\") pod \"ovnkube-node-dwcm5\" (UID: \"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353\") " pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.015452 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.015377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8nz\" (UniqueName: \"kubernetes.io/projected/6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e-kube-api-access-wn8nz\") pod \"multus-4nbss\" (UID: \"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e\") " pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.015452 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.015402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw47r\" (UniqueName: \"kubernetes.io/projected/d9722643-4aff-48fd-8bef-8abe70c56b18-kube-api-access-tw47r\") pod \"iptables-alerter-ktn7p\" (UID: \"d9722643-4aff-48fd-8bef-8abe70c56b18\") " pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.015548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.015458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk44x\" (UniqueName: \"kubernetes.io/projected/5082e209-4074-4742-9da7-0ce1bb74779f-kube-api-access-kk44x\") pod \"node-ca-d8j6c\" (UID: \"5082e209-4074-4742-9da7-0ce1bb74779f\") " pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.028602 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.028548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal" event={"ID":"d77a65267e4463f8e5a16dfb51a44592","Type":"ContainerStarted","Data":"725eb178346b90643608ebdfed9121278c238128179a76ceb714269c917c8967"}
Aug 04 18:00:31.029540 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.029517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal" event={"ID":"a17c8ab47d99a99d58a14551ab35cc2c","Type":"ContainerStarted","Data":"727dd57d88e13e8a499e155920d55359d6ac59009062f83650d3bf3655167f24"}
Aug 04 18:00:31.107081 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-modprobe-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107081 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-etc-selinux\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-tmp\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbdf\" (UniqueName: \"kubernetes.io/projected/126a405e-09cc-479a-acfa-68a3b51990ff-kube-api-access-mpbdf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkshf\" (UniqueName: \"kubernetes.io/projected/2550b4e7-088e-48e4-abc9-2a500311fabc-kube-api-access-qkshf\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysconfig\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-run\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-host\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-modprobe-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-etc-selinux\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21136c2d-8c4a-41d9-9b45-9d55c3599539-hosts-file\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysconfig\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-run\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21136c2d-8c4a-41d9-9b45-9d55c3599539-hosts-file\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-var-lib-kubelet\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-host\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.107390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-system-cni-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-socket-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-var-lib-kubelet\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-system-cni-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-sys\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-os-release\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-sys\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-socket-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvt7\" (UniqueName: \"kubernetes.io/projected/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kube-api-access-hqvt7\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gkj\" (UniqueName: \"kubernetes.io/projected/6642e299-8f13-49a5-835f-02c797ab231e-kube-api-access-z4gkj\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-os-release\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-kubernetes\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-conf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-registration-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.108266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-kubernetes\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-device-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-sys-fs\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-registration-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed50106d-71e8-4606-a3fd-081ac58e180c-konnectivity-ca\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-device-dir\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-systemd\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.107991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-conf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-systemd\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108012    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-sys-fs\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-etc-tuned\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed50106d-71e8-4606-a3fd-081ac58e180c-agent-certs\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-cnibin\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109043 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.108119    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.108177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.608159551 +0000 UTC m=+3.079446809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-lib-modules\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21136c2d-8c4a-41d9-9b45-9d55c3599539-tmp-dir\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-etc-sysctl-d\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrg2\" (UniqueName: \"kubernetes.io/projected/21136c2d-8c4a-41d9-9b45-9d55c3599539-kube-api-access-snrg2\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6642e299-8f13-49a5-835f-02c797ab231e-cnibin\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/126a405e-09cc-479a-acfa-68a3b51990ff-lib-modules\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21136c2d-8c4a-41d9-9b45-9d55c3599539-tmp-dir\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.109661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.108878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6642e299-8f13-49a5-835f-02c797ab231e-cni-binary-copy\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.110093 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.109736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-tmp\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.110093 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.109758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed50106d-71e8-4606-a3fd-081ac58e180c-konnectivity-ca\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.110525 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.110498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/126a405e-09cc-479a-acfa-68a3b51990ff-etc-tuned\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.110924 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.110903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed50106d-71e8-4606-a3fd-081ac58e180c-agent-certs\") pod \"konnectivity-agent-9fwpg\" (UID: \"ed50106d-71e8-4606-a3fd-081ac58e180c\") " pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.117765 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.117738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkshf\" (UniqueName: \"kubernetes.io/projected/2550b4e7-088e-48e4-abc9-2a500311fabc-kube-api-access-qkshf\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.117886 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.117857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gkj\" (UniqueName: \"kubernetes.io/projected/6642e299-8f13-49a5-835f-02c797ab231e-kube-api-access-z4gkj\") pod \"multus-additional-cni-plugins-4ktf9\" (UID: \"6642e299-8f13-49a5-835f-02c797ab231e\") " pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.118037 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.118017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbdf\" (UniqueName: \"kubernetes.io/projected/126a405e-09cc-479a-acfa-68a3b51990ff-kube-api-access-mpbdf\") pod \"tuned-lz22x\" (UID: \"126a405e-09cc-479a-acfa-68a3b51990ff\") " pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.118786 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.118766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrg2\" (UniqueName: \"kubernetes.io/projected/21136c2d-8c4a-41d9-9b45-9d55c3599539-kube-api-access-snrg2\") pod \"node-resolver-t8vf5\" (UID: \"21136c2d-8c4a-41d9-9b45-9d55c3599539\") " pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.118828 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.118782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvt7\" (UniqueName: \"kubernetes.io/projected/e6c019c4-800f-4e0f-bbdc-a03124b92cfc-kube-api-access-hqvt7\") pod \"aws-ebs-csi-driver-node-89m2v\" (UID: \"e6c019c4-800f-4e0f-bbdc-a03124b92cfc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.195089 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.195056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d8j6c"
Aug 04 18:00:31.202924 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.202903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nbss"
Aug 04 18:00:31.204366 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.204330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5082e209_4074_4742_9da7_0ce1bb74779f.slice/crio-ed547fb21b584fe3db23be53f117a1f895c1d83e128fcaed950a42f55e0e2e24 WatchSource:0}: Error finding container ed547fb21b584fe3db23be53f117a1f895c1d83e128fcaed950a42f55e0e2e24: Status 404 returned error can't find the container with id ed547fb21b584fe3db23be53f117a1f895c1d83e128fcaed950a42f55e0e2e24
Aug 04 18:00:31.210661 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.210637    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbd87d7_6707_4dd7_aa6b_2d3a8fe85a8e.slice/crio-0a64fbc91371ca540664c964afc895ba415c9e255546b43dcc3257180999b511 WatchSource:0}: Error finding container 0a64fbc91371ca540664c964afc895ba415c9e255546b43dcc3257180999b511: Status 404 returned error can't find the container with id 0a64fbc91371ca540664c964afc895ba415c9e255546b43dcc3257180999b511
Aug 04 18:00:31.210746 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.210672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ktn7p"
Aug 04 18:00:31.216121 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.216082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:31.218446 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.218307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9722643_4aff_48fd_8bef_8abe70c56b18.slice/crio-babb71fedb31fbdd47d42dcede3051886cda5d8affdee6196e03d8ee31c7409d WatchSource:0}: Error finding container babb71fedb31fbdd47d42dcede3051886cda5d8affdee6196e03d8ee31c7409d: Status 404 returned error can't find the container with id babb71fedb31fbdd47d42dcede3051886cda5d8affdee6196e03d8ee31c7409d
Aug 04 18:00:31.221170 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.221155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v"
Aug 04 18:00:31.223892 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.223863    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0ebca0_e0b5_47d3_8f9e_0ec3a22e1353.slice/crio-d57e84c15d28f0c264b86d2fd6b1a1e7cef115461b0be0b36199051d592f530a WatchSource:0}: Error finding container d57e84c15d28f0c264b86d2fd6b1a1e7cef115461b0be0b36199051d592f530a: Status 404 returned error can't find the container with id d57e84c15d28f0c264b86d2fd6b1a1e7cef115461b0be0b36199051d592f530a
Aug 04 18:00:31.226695 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.226678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lz22x"
Aug 04 18:00:31.229687 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.229667    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c019c4_800f_4e0f_bbdc_a03124b92cfc.slice/crio-cce56c48d52fcc3f8c7e130ce35fa48f4387d42957ed5d421af06a12affdce32 WatchSource:0}: Error finding container cce56c48d52fcc3f8c7e130ce35fa48f4387d42957ed5d421af06a12affdce32: Status 404 returned error can't find the container with id cce56c48d52fcc3f8c7e130ce35fa48f4387d42957ed5d421af06a12affdce32
Aug 04 18:00:31.233718 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.232813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8vf5"
Aug 04 18:00:31.236842 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.236813    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126a405e_09cc_479a_acfa_68a3b51990ff.slice/crio-e68c5fa64c7d2f113be830e92141fb6bbb72621b970e4df6eebc1eb43fde0448 WatchSource:0}: Error finding container e68c5fa64c7d2f113be830e92141fb6bbb72621b970e4df6eebc1eb43fde0448: Status 404 returned error can't find the container with id e68c5fa64c7d2f113be830e92141fb6bbb72621b970e4df6eebc1eb43fde0448
Aug 04 18:00:31.238440 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.238419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:31.241180 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.241154    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21136c2d_8c4a_41d9_9b45_9d55c3599539.slice/crio-621fc97525e96996e228f0f0fbca71afbb67af8248581bd88cd0653fdf337890 WatchSource:0}: Error finding container 621fc97525e96996e228f0f0fbca71afbb67af8248581bd88cd0653fdf337890: Status 404 returned error can't find the container with id 621fc97525e96996e228f0f0fbca71afbb67af8248581bd88cd0653fdf337890
Aug 04 18:00:31.243246 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.243224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4ktf9"
Aug 04 18:00:31.246739 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.246718    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded50106d_71e8_4606_a3fd_081ac58e180c.slice/crio-99fb359a7c45b85a77a53db4116e366282b871a1b45484ed62d5211df2de4020 WatchSource:0}: Error finding container 99fb359a7c45b85a77a53db4116e366282b871a1b45484ed62d5211df2de4020: Status 404 returned error can't find the container with id 99fb359a7c45b85a77a53db4116e366282b871a1b45484ed62d5211df2de4020
Aug 04 18:00:31.252139 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:00:31.252117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6642e299_8f13_49a5_835f_02c797ab231e.slice/crio-d53d9202ad9a4583061f2236d6a3e09814bfa21d6fa99c10165bcc8493568a9a WatchSource:0}: Error finding container d53d9202ad9a4583061f2236d6a3e09814bfa21d6fa99c10165bcc8493568a9a: Status 404 returned error can't find the container with id d53d9202ad9a4583061f2236d6a3e09814bfa21d6fa99c10165bcc8493568a9a
Aug 04 18:00:31.314644 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.314605    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:31.510909 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.510819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:31.511079 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.510959    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.511079 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.511031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.511011691 +0000 UTC m=+3.982298944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.611771 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.611735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:31.611965 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.611809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:31.612032 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.611980    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:31.612032 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.611999    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:31.612032 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.612013    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.612191 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.612076    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.61205631 +0000 UTC m=+4.083343567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.612529 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.612502    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:31.612604 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:31.612563    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:00:32.612547327 +0000 UTC m=+4.083834587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:31.938839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.938744    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:31 +0000 UTC" deadline="2028-03-20 07:42:12.323515797 +0000 UTC"
Aug 04 18:00:31.938839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:31.938782    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14245h41m40.384738628s"
Aug 04 18:00:32.025149 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.025117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:32.025314 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.025246    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:32.033407 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.033372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fwpg" event={"ID":"ed50106d-71e8-4606-a3fd-081ac58e180c","Type":"ContainerStarted","Data":"99fb359a7c45b85a77a53db4116e366282b871a1b45484ed62d5211df2de4020"}
Aug 04 18:00:32.035148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.035121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lz22x" event={"ID":"126a405e-09cc-479a-acfa-68a3b51990ff","Type":"ContainerStarted","Data":"e68c5fa64c7d2f113be830e92141fb6bbb72621b970e4df6eebc1eb43fde0448"}
Aug 04 18:00:32.037261 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.037235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v" event={"ID":"e6c019c4-800f-4e0f-bbdc-a03124b92cfc","Type":"ContainerStarted","Data":"cce56c48d52fcc3f8c7e130ce35fa48f4387d42957ed5d421af06a12affdce32"}
Aug 04 18:00:32.039109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.039086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"d57e84c15d28f0c264b86d2fd6b1a1e7cef115461b0be0b36199051d592f530a"}
Aug 04 18:00:32.040445 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.040422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ktn7p" event={"ID":"d9722643-4aff-48fd-8bef-8abe70c56b18","Type":"ContainerStarted","Data":"babb71fedb31fbdd47d42dcede3051886cda5d8affdee6196e03d8ee31c7409d"}
Aug 04 18:00:32.042955 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.042933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nbss" event={"ID":"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e","Type":"ContainerStarted","Data":"0a64fbc91371ca540664c964afc895ba415c9e255546b43dcc3257180999b511"}
Aug 04 18:00:32.044683 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.044660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d8j6c" event={"ID":"5082e209-4074-4742-9da7-0ce1bb74779f","Type":"ContainerStarted","Data":"ed547fb21b584fe3db23be53f117a1f895c1d83e128fcaed950a42f55e0e2e24"}
Aug 04 18:00:32.046099 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.046073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerStarted","Data":"d53d9202ad9a4583061f2236d6a3e09814bfa21d6fa99c10165bcc8493568a9a"}
Aug 04 18:00:32.047426 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.047402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8vf5" event={"ID":"21136c2d-8c4a-41d9-9b45-9d55c3599539","Type":"ContainerStarted","Data":"621fc97525e96996e228f0f0fbca71afbb67af8248581bd88cd0653fdf337890"}
Aug 04 18:00:32.519567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.519526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:32.519739 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.519682    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:32.519799 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.519761    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.519739536 +0000 UTC m=+5.991026794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:32.621046 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.620989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:32.621233 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.621068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:32.621233 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621215    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:32.621356 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621236    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:32.621356 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621250    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:32.621356 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.621289289 +0000 UTC m=+6.092576557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:32.621775 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621749    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:32.621882 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:32.621816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.621796242 +0000 UTC m=+6.093083510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:32.940016 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.939956    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:31 +0000 UTC" deadline="2028-03-05 12:02:43.099473366 +0000 UTC"
Aug 04 18:00:32.940016 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:32.940017    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13890h2m10.159460162s"
Aug 04 18:00:33.027417 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:33.027390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:33.027587 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:33.027528    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:33.027669 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:33.027607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:33.027726 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:33.027679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:34.025398 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:34.025369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:34.025876 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.025492    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:34.064380 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:34.064342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal" event={"ID":"a17c8ab47d99a99d58a14551ab35cc2c","Type":"ContainerStarted","Data":"083e09cff41a2a0733263912595b60398bf730f605baba9853d733435188a8d2"}
Aug 04 18:00:34.535571 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:34.535537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:34.535744 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.535726    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.535814 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.535781    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.535763388 +0000 UTC m=+10.007050645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:34.636126 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:34.636091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:34.636303 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:34.636188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:34.636368 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636329    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:34.636431 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.636367711 +0000 UTC m=+10.107654963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:34.636767 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636639    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:34.636767 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636661    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:34.636767 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636674    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:34.636767 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:34.636728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:38.636704387 +0000 UTC m=+10.107991642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:35.025275 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:35.025199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:35.025442 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:35.025207    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:35.025442 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:35.025335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:35.025442 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:35.025417    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:36.024707 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:36.024663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:36.024903 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:36.024795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:37.025451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:37.024924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:37.025451 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:37.025054    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:37.025959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:37.025548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:37.025959 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:37.025679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:37.073892 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:37.073839    2579 generic.go:356] "Generic (PLEG): container finished" podID="a17c8ab47d99a99d58a14551ab35cc2c" containerID="083e09cff41a2a0733263912595b60398bf730f605baba9853d733435188a8d2" exitCode=0
Aug 04 18:00:37.073892 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:37.073867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal" event={"ID":"a17c8ab47d99a99d58a14551ab35cc2c","Type":"ContainerDied","Data":"083e09cff41a2a0733263912595b60398bf730f605baba9853d733435188a8d2"}
Aug 04 18:00:38.025651 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:38.025547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:38.026082 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.025673    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:38.569398 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:38.569334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:38.569611 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.569468    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:38.569611 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.569563    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.569542415 +0000 UTC m=+18.040829669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:38.669691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:38.669761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.669903    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.669925    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.669938    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.669996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.66997926 +0000 UTC m=+18.141266513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.670031    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:38.670170 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:38.670069    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:00:46.670057801 +0000 UTC m=+18.141345066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:39.026509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:39.026012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:39.026509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:39.026095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:39.026509 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:39.026120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:39.026509 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:39.026217    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:40.025526 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:40.025133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:40.025526 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:40.025276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:41.024733 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:41.024696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:41.025168 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:41.024832    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:41.025168 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:41.024886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:41.025667 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:41.025629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:42.025364 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:42.025331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:42.025826 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:42.025455    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:43.025048 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:43.025020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:43.025238 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:43.025157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:43.025238 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:43.025216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:43.025366 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:43.025343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:44.025023 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:44.024981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:44.025459 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:44.025126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:45.024713 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:45.024675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:45.024902 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:45.024807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:45.024902 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:45.024867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:45.025007 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:45.024969    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:46.024630 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:46.024597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:46.025002 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.024717    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:46.635283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:46.635251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:46.635508 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.635412    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:46.635508 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.635483    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.635461008 +0000 UTC m=+34.106748263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:46.735756 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:46.735714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:46.735779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735835    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735878    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735889    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.735874009 +0000 UTC m=+34.207161260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735892    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:46.735916 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735901    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:46.736239 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:46.735937    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.735927277 +0000 UTC m=+34.207214529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:47.025547 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:47.025457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:47.025547 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:47.025524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:47.026017 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:47.025622    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:47.026017 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:47.025768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:48.025592 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:48.025558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:48.026051 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:48.025687    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:49.025929 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:49.025876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:49.026410 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:49.025985    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:49.026410 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:49.026059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:49.026410 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:49.026193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:50.024930 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:50.024893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:50.025097 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:50.025032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:51.025340 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.025297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:51.025340 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.025342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:51.026338 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:51.025458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:51.026338 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:51.025577    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:51.099704 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.099674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal" event={"ID":"a17c8ab47d99a99d58a14551ab35cc2c","Type":"ContainerStarted","Data":"f6375f608ea729d6969b1ef5a14e5f799a0329482bd987f56bc738cc0d5569db"}
Aug 04 18:00:51.101341 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.101311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9fwpg" event={"ID":"ed50106d-71e8-4606-a3fd-081ac58e180c","Type":"ContainerStarted","Data":"d0456e02fd7f8713a6aa7ffcd4825b242817821ab3435439af42161b8c4d7612"}
Aug 04 18:00:51.102812 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.102793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lz22x" event={"ID":"126a405e-09cc-479a-acfa-68a3b51990ff","Type":"ContainerStarted","Data":"6cfe4e247264c387f7fe850fb0c334b37934fcf9e6b2073bb4b57bf74ad5463a"}
Aug 04 18:00:51.104900 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.104880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v" event={"ID":"e6c019c4-800f-4e0f-bbdc-a03124b92cfc","Type":"ContainerStarted","Data":"d83312a4432ff0821d0a54645dcac9477e4dd7f608fa596246157dbfe8e690f7"}
Aug 04 18:00:51.107791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.107773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:00:51.108115 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108097    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353" containerID="d7cb867accb1b610138f0c95ae646f988a4c93973e9724458d154c32fac936a7" exitCode=1
Aug 04 18:00:51.108159 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"cd66c6d4c3c976332e804b4dabd7d6b1b9fc6c6ee1cfaa27c7444b0965efaba6"}
Aug 04 18:00:51.108228 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"9abb829fa0521b5958a95b144f7be1f420901a31d18e4e9f746547fc8a500e29"}
Aug 04 18:00:51.108293 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"786c4aa5fce65c74c6814437f49be582faf25724d82fa05f60e0ac540dd50cec"}
Aug 04 18:00:51.108293 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"da50d714f1f8d4695803a1c44dbfc61357323120c63a756cafcd16fa904485a6"}
Aug 04 18:00:51.108370 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerDied","Data":"d7cb867accb1b610138f0c95ae646f988a4c93973e9724458d154c32fac936a7"}
Aug 04 18:00:51.108370 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.108350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"2fea423cd69cbbc92ce3f8095808db2ac21d88b3d5b197b786e2e8dfc216634c"}
Aug 04 18:00:51.109587 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.109567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nbss" event={"ID":"6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e","Type":"ContainerStarted","Data":"367efbb02918b6fd35cf4c9c2c7e6cc21acdb433ea709c353db8d34262910af6"}
Aug 04 18:00:51.111444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.111421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d8j6c" event={"ID":"5082e209-4074-4742-9da7-0ce1bb74779f","Type":"ContainerStarted","Data":"cfd2f4dfafc46a50586c053e2b67fae9927e8020abdc61bc0eb50ee758d06c6d"}
Aug 04 18:00:51.112988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.112969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal" event={"ID":"d77a65267e4463f8e5a16dfb51a44592","Type":"ContainerStarted","Data":"7fb21862481b9562a35e9e45617e1c39d3e8bfa5926d22239852aaf9e45cfe4a"}
Aug 04 18:00:51.114590 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.114565    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="4883758ff4396683a71ca316439a2f32f32573a404d214481ccdcf477334156f" exitCode=0
Aug 04 18:00:51.114670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.114635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"4883758ff4396683a71ca316439a2f32f32573a404d214481ccdcf477334156f"}
Aug 04 18:00:51.116326 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.116307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8vf5" event={"ID":"21136c2d-8c4a-41d9-9b45-9d55c3599539","Type":"ContainerStarted","Data":"48daad4681c19b14766e3dd721d93c49b6595ff875bd297478811c65a4566d7f"}
Aug 04 18:00:51.117473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.117350    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-250.ec2.internal" podStartSLOduration=21.117338934 podStartE2EDuration="21.117338934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.11678601 +0000 UTC m=+22.588073302" watchObservedRunningTime="2026-08-04 18:00:51.117338934 +0000 UTC m=+22.588626207"
Aug 04 18:00:51.135786 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.135743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lz22x" podStartSLOduration=3.08891434 podStartE2EDuration="22.135732198s" totalImagesPullingTime="19.046817858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.135146035 +0000 UTC m=+22.606433334" watchObservedRunningTime="2026-08-04 18:00:51.135732198 +0000 UTC m=+22.607019471"
Aug 04 18:00:51.179424 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.179373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8vf5" podStartSLOduration=7.375943946 podStartE2EDuration="22.179359182s" totalImagesPullingTime="14.803415236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.151229356 +0000 UTC m=+22.622516629" watchObservedRunningTime="2026-08-04 18:00:51.179359182 +0000 UTC m=+22.650646455"
Aug 04 18:00:51.194102 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.194061    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-250.ec2.internal" podStartSLOduration=21.194047997 podStartE2EDuration="21.194047997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.194041716 +0000 UTC m=+22.665328990" watchObservedRunningTime="2026-08-04 18:00:51.194047997 +0000 UTC m=+22.665335270"
Aug 04 18:00:51.228550 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.228477    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d8j6c" podStartSLOduration=3.163514692 podStartE2EDuration="22.228456176s" totalImagesPullingTime="19.064941484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.209889688 +0000 UTC m=+22.681176962" watchObservedRunningTime="2026-08-04 18:00:51.228456176 +0000 UTC m=+22.699743451"
Aug 04 18:00:51.228688 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.228640    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4nbss" podStartSLOduration=2.618399629 podStartE2EDuration="22.228632043s" totalImagesPullingTime="19.610232414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.228367178 +0000 UTC m=+22.699654470" watchObservedRunningTime="2026-08-04 18:00:51.228632043 +0000 UTC m=+22.699919319"
Aug 04 18:00:51.248670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.248626    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9fwpg" podStartSLOduration=3.226760538 podStartE2EDuration="22.248611226s" totalImagesPullingTime="19.021850688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.248471103 +0000 UTC m=+22.719758377" watchObservedRunningTime="2026-08-04 18:00:51.248611226 +0000 UTC m=+22.719898500"
Aug 04 18:00:51.840995 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.840962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:51.841773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.841750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:51.919587 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.919562    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:51.953959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.953853    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:51.919579788Z","UUID":"19dc1272-f7c3-4e21-8b8d-c64e69f431af","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:51.957202 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.957181    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:51.957308 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:51.957226    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:52.024825 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:52.024798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:52.024998 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:52.024901    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:52.121003 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:52.120971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v" event={"ID":"e6c019c4-800f-4e0f-bbdc-a03124b92cfc","Type":"ContainerStarted","Data":"6aa0fca133a59aaef949d29069a34b50f37de51a24970eac8cc1313df82a100b"}
Aug 04 18:00:52.122563 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:52.122531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ktn7p" event={"ID":"d9722643-4aff-48fd-8bef-8abe70c56b18","Type":"ContainerStarted","Data":"0433a25c1f266b7229bcdf962b3d8ca6e142b6e7c2dd3323affd2aa689a24a00"}
Aug 04 18:00:53.025311 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:53.025281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:53.025529 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:53.025413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:53.025529 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:53.025462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:53.025649 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:53.025593    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:53.128793 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:53.128769    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:00:53.129295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:53.129255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"81343fc53dd9caaf89b448aab47a7a1bd7462c5ea5955bd66c15986a8de4a0c0"}
Aug 04 18:00:53.129295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:53.129284    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 04 18:00:54.025165 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:54.024987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:54.025326 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:54.025259    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:54.137104 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:54.137065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v" event={"ID":"e6c019c4-800f-4e0f-bbdc-a03124b92cfc","Type":"ContainerStarted","Data":"cff6b3d4389c9497e942a9aa9a7aa590dc6fc9612024f24c5f7ac8316bface86"}
Aug 04 18:00:54.161191 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:54.161136    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ktn7p" podStartSLOduration=10.335054119 podStartE2EDuration="25.161116699s" totalImagesPullingTime="14.82606258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:52.140701536 +0000 UTC m=+23.611988813" watchObservedRunningTime="2026-08-04 18:00:54.161116699 +0000 UTC m=+25.632403977"
Aug 04 18:00:55.025720 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:55.025690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:55.025882 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:55.025818    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:55.025882 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:55.025852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:55.025957 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:55.025908    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:56.025563 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.025376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:56.026244 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:56.025632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:56.144241 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.144215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:00:56.144563 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.144541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"80f42087889088063ab992d236ca3c74dbc1f139669668811a6de121ab9f6988"}
Aug 04 18:00:56.144885 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.144854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:56.144972 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.144952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:56.145101 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.145031    2579 scope.go:117] "RemoveContainer" containerID="d7cb867accb1b610138f0c95ae646f988a4c93973e9724458d154c32fac936a7"
Aug 04 18:00:56.146374 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.146351    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="dfb49fc0b9cbf8ff4d792bcd7d1b03833c2fd8e137d32c0da5b64136e7b88f86" exitCode=0
Aug 04 18:00:56.146468 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.146382    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"dfb49fc0b9cbf8ff4d792bcd7d1b03833c2fd8e137d32c0da5b64136e7b88f86"}
Aug 04 18:00:56.161414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.161393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:56.178970 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:56.178922    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89m2v" podStartSLOduration=5.346724546 podStartE2EDuration="27.178909312s" totalImagesPullingTime="21.832184766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:54.161097437 +0000 UTC m=+25.632384752" watchObservedRunningTime="2026-08-04 18:00:56.178909312 +0000 UTC m=+27.650196627"
Aug 04 18:00:57.025284 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.025256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:57.025524 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:57.025355    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:57.025524 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.025405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:57.025930 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:57.025520    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:57.151210 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.151185    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:00:57.151536 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.151512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" event={"ID":"8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353","Type":"ContainerStarted","Data":"40c8b5502a332726318d298e861e36a1a2ea0dd8df980aceb13cba93e7c9d92d"}
Aug 04 18:00:57.151721 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.151705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:57.166118 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.166098    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:00:57.187508 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.187459    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5" podStartSLOduration=9.076554713 podStartE2EDuration="28.187445555s" totalImagesPullingTime="19.110890842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:57.18628915 +0000 UTC m=+28.657576434" watchObservedRunningTime="2026-08-04 18:00:57.187445555 +0000 UTC m=+28.658732811"
Aug 04 18:00:57.841464 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.841399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:57.841600 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.841542    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Aug 04 18:00:57.841928 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.841906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9fwpg"
Aug 04 18:00:57.899992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.899964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5ldx"]
Aug 04 18:00:57.900111 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.900062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:00:57.900161 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:57.900138    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:00:57.904243 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.904221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw5pt"]
Aug 04 18:00:57.904338 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.904309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:00:57.904412 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:57.904394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:00:57.908253 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.908233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tglfd"]
Aug 04 18:00:57.908321 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:57.908294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:00:57.908367 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:00:57.908356    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:00:58.155930 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:58.155901    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="b8c55966a0a5d8b1c8319967158f1ffaec7ef2d94d221af5d5ab55836bef7b27" exitCode=0
Aug 04 18:00:58.156306 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:00:58.155983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"b8c55966a0a5d8b1c8319967158f1ffaec7ef2d94d221af5d5ab55836bef7b27"}
Aug 04 18:01:00.025292 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:00.025268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:00.025593 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:00.025268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:01:00.025593 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:00.025368    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:01:00.025593 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:00.025450    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:01:00.025593 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:00.025268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:00.025593 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:00.025543    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:01:01.165123 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:01.164951    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="af11699a31831dbfa013726d749e1b8417cc367fbf2d38572915157cbdfdd74e" exitCode=0
Aug 04 18:01:01.165123 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:01.165026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"af11699a31831dbfa013726d749e1b8417cc367fbf2d38572915157cbdfdd74e"}
Aug 04 18:01:02.024915 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.024824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:02.025102 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.024948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:01:02.025102 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.024949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tglfd" podUID="56060bb1-3104-4bbb-beb9-008c28976663"
Aug 04 18:01:02.025102 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.024996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:02.025102 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.025089    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jw5pt" podUID="2550b4e7-088e-48e4-abc9-2a500311fabc"
Aug 04 18:01:02.025255 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.025157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m5ldx" podUID="7d06c764-6c99-4554-a19c-b8f08450baef"
Aug 04 18:01:02.370472 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.370449    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-250.ec2.internal" event="NodeReady"
Aug 04 18:01:02.371012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.370600    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:01:02.406487 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.406458    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:01:02.408659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.408637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.411867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.411622    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9xbrz\""
Aug 04 18:01:02.411867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.411652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 18:01:02.411867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.411652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 18:01:02.411867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.411655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 18:01:02.418280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.418259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 18:01:02.421899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.421878    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bbmqg"]
Aug 04 18:01:02.424303 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.424286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:01:02.424421 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.424380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.424793 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.424609    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6dvr"]
Aug 04 18:01:02.426992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.426743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nwtj6\""
Aug 04 18:01:02.426992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.426842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:01:02.426992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.426745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:01:02.427734 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.427659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.430062 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.430043    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwpcl\""
Aug 04 18:01:02.430146 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.430059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:01:02.430365 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.430345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:01:02.430449 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.430345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:01:02.439249 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.439226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6dvr"]
Aug 04 18:01:02.439324 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.439309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbmqg"]
Aug 04 18:01:02.561034 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561177 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwv7\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561177 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-cert\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.561177 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561177 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/008642e1-a70d-496a-876c-e70fc8b02be0-config-volume\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr478\" (UniqueName: \"kubernetes.io/projected/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-kube-api-access-tr478\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4ck\" (UniqueName: \"kubernetes.io/projected/008642e1-a70d-496a-876c-e70fc8b02be0-kube-api-access-dp4ck\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.561480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.561425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008642e1-a70d-496a-876c-e70fc8b02be0-tmp-dir\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.591480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.591443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:02.662548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwv7\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:02.662701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-cert\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.662701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/008642e1-a70d-496a-876c-e70fc8b02be0-config-volume\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr478\" (UniqueName: \"kubernetes.io/projected/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-kube-api-access-tr478\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.662896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4ck\" (UniqueName: \"kubernetes.io/projected/008642e1-a70d-496a-876c-e70fc8b02be0-kube-api-access-dp4ck\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.662957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008642e1-a70d-496a-876c-e70fc8b02be0-tmp-dir\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.662828    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663080    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:03.163057185 +0000 UTC m=+34.634344436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663081    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:02.663161 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663109    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:02.663477 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663171    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:03.163154767 +0000 UTC m=+34.634442020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:02.663477 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663023    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:01:02.663477 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.663322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008642e1-a70d-496a-876c-e70fc8b02be0-tmp-dir\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.663743 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.663720    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret podName:7d06c764-6c99-4554-a19c-b8f08450baef nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.663699154 +0000 UTC m=+66.134986428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret") pod "global-pull-secret-syncer-m5ldx" (UID: "7d06c764-6c99-4554-a19c-b8f08450baef") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:01:02.663883 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.663760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.663939 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.663917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/008642e1-a70d-496a-876c-e70fc8b02be0-config-volume\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.664066 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.664042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.664172 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.664152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.667690 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.667667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.667787 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.667687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.667787 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.667707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-cert\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.673547 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.673194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwv7\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.673547 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.673202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr478\" (UniqueName: \"kubernetes.io/projected/d39c81e0-45c6-4b3e-beb7-d9795902a1e8-kube-api-access-tr478\") pod \"ingress-canary-j6dvr\" (UID: \"d39c81e0-45c6-4b3e-beb7-d9795902a1e8\") " pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.673547 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.673521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4ck\" (UniqueName: \"kubernetes.io/projected/008642e1-a70d-496a-876c-e70fc8b02be0-kube-api-access-dp4ck\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:02.674515 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.674470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:02.743731 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.743711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6dvr"
Aug 04 18:01:02.763786 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.763763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:01:02.763868 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.763828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:02.763945 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.763920    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:01:02.763978 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.763957    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:01:02.763978 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.763971    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:01:02.764054 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.763984    2579 projected.go:196] Error preparing data for projected volume kube-api-access-rrq77 for pod openshift-network-diagnostics/network-check-target-tglfd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:01:02.764054 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.764018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.763992411 +0000 UTC m=+66.235279690 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:01:02.764054 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:02.764044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77 podName:56060bb1-3104-4bbb-beb9-008c28976663 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.764031247 +0000 UTC m=+66.235318499 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrq77" (UniqueName: "kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77") pod "network-check-target-tglfd" (UID: "56060bb1-3104-4bbb-beb9-008c28976663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:01:02.892880 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:02.892838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6dvr"]
Aug 04 18:01:02.899083 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:02.899047    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39c81e0_45c6_4b3e_beb7_d9795902a1e8.slice/crio-592b43fdcf47a9b6582de6ebc849a0ed7a69b50b2b2dd0db335717799f7b2591 WatchSource:0}: Error finding container 592b43fdcf47a9b6582de6ebc849a0ed7a69b50b2b2dd0db335717799f7b2591: Status 404 returned error can't find the container with id 592b43fdcf47a9b6582de6ebc849a0ed7a69b50b2b2dd0db335717799f7b2591
Aug 04 18:01:03.167511 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:03.167322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:03.167665 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:03.167556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:03.167665 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:03.167485    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:03.167665 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:03.167655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:04.167633935 +0000 UTC m=+35.638921190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:03.167814 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:03.167682    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:03.167814 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:03.167696    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:03.167814 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:03.167735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:04.167724689 +0000 UTC m=+35.639011941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:03.170969 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:03.170932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6dvr" event={"ID":"d39c81e0-45c6-4b3e-beb7-d9795902a1e8","Type":"ContainerStarted","Data":"592b43fdcf47a9b6582de6ebc849a0ed7a69b50b2b2dd0db335717799f7b2591"}
Aug 04 18:01:03.559920 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:03.559846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:03.615794 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:03.615766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.024645 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.024618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:01:04.024645 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.024641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:04.024881 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.024847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:04.027926 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.027908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:04.028662 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.028642    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:04.028763 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.028741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:04.028820 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.028765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x566h\""
Aug 04 18:01:04.028820 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.028777    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bbhlr\""
Aug 04 18:01:04.028929 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.028861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:04.041775 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.041753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.145858 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.145828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.176457 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.176430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:04.176586 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:04.176543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:04.176648 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:04.176597    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:04.176648 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:04.176616    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:04.176759 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:04.176651    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:04.176759 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:04.176686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.176666158 +0000 UTC m=+37.647953412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:04.176759 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:04.176705    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.176693335 +0000 UTC m=+37.647980586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:05.441892 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:05.441817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:06.181012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:06.180978    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="64d00ea13aac9e55f71b94325da00c4a40a49a83d0e1c9898694af4eca377be1" exitCode=0
Aug 04 18:01:06.181148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:06.181054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"64d00ea13aac9e55f71b94325da00c4a40a49a83d0e1c9898694af4eca377be1"}
Aug 04 18:01:06.192817 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:06.192793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:06.192937 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:06.192871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:06.192937 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:06.192904    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:06.193005 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:06.192960    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:06.193005 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:06.192968    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.192952938 +0000 UTC m=+41.664240190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:06.193005 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:06.192970    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:06.193005 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:06.192997    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.192991087 +0000 UTC m=+41.664278338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:07.185946 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:07.185916    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642e299-8f13-49a5-835f-02c797ab231e" containerID="8a39d9106a38fba8684a960ad6f6091bb86e6071292736cb842840295adb9040" exitCode=0
Aug 04 18:01:07.186348 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:07.185959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerDied","Data":"8a39d9106a38fba8684a960ad6f6091bb86e6071292736cb842840295adb9040"}
Aug 04 18:01:08.191283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.191090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" event={"ID":"6642e299-8f13-49a5-835f-02c797ab231e","Type":"ContainerStarted","Data":"b1d9efe9d6baba5cef74ac5f81b306d089c84c6ac42ba8267a74ca6fc66fa4b5"}
Aug 04 18:01:08.192566 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.192542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6dvr" event={"ID":"d39c81e0-45c6-4b3e-beb7-d9795902a1e8","Type":"ContainerStarted","Data":"c27cc8e1104e9910ffe085238352ecf64202d2fc57f39c043842c8eea005204d"}
Aug 04 18:01:08.220323 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.220286    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4ktf9" podStartSLOduration=8.229902692 podStartE2EDuration="39.22027242s" totalImagesPullingTime="30.990369728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.21979746 +0000 UTC m=+39.691084735" watchObservedRunningTime="2026-08-04 18:01:08.22027242 +0000 UTC m=+39.691559694"
Aug 04 18:01:08.362918 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.362888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.427797 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.427771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.650177 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.650129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:08.715873 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:08.715844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.225602 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:10.225564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:10.225617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:10.225710    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:10.225711    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:10.225787    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.22577195 +0000 UTC m=+49.697059202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:10.225720    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:10.225976 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:10.225853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.22583801 +0000 UTC m=+49.697125266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:14.857766 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:14.857732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:14.960139 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:14.960109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.832939 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.832879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6dvr" podStartSLOduration=11.526925778 podStartE2EDuration="15.8328608s" totalImagesPullingTime="4.305935022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:08.237392037 +0000 UTC m=+39.708679311" watchObservedRunningTime="2026-08-04 18:01:17.8328608 +0000 UTC m=+49.304148072"
Aug 04 18:01:17.834005 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.833980    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"]
Aug 04 18:01:17.883864 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.883840    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-wwdm7"]
Aug 04 18:01:17.883969 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.883948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:17.888147 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.888128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Aug 04 18:01:17.890232 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.890085    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hrp4p\""
Aug 04 18:01:17.890232 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.890100    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Aug 04 18:01:17.890232 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.890139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:01:17.890232 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.890176    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:01:17.903283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.903263    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-pglq2"]
Aug 04 18:01:17.903388 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.903372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:17.907436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.907417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Aug 04 18:01:17.907555 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.907449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Aug 04 18:01:17.908254 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.908234    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:01:17.908533 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.908517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qcrzn\""
Aug 04 18:01:17.908607 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.908520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:01:17.913652 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.913632    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f6d8894dd-p9sxx"]
Aug 04 18:01:17.913744 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.913730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:17.920359 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.920242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Aug 04 18:01:17.920954 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.920661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 18:01:17.920954 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.920813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 18:01:17.920954 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.920857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-29f5z\""
Aug 04 18:01:17.921750 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.921665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 18:01:17.924914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.924894    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 18:01:17.925945 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.925928    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:17.935595 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.935578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"]
Aug 04 18:01:17.935669 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.935603    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-pglq2"]
Aug 04 18:01:17.935669 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.935615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wwdm7"]
Aug 04 18:01:17.935669 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.935626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6d8894dd-p9sxx"]
Aug 04 18:01:17.935758 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.935684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:17.939000 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.938973    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Aug 04 18:01:17.939105 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.938976    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 18:01:17.939105 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.939014    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Aug 04 18:01:17.939105 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.939094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h78mq\""
Aug 04 18:01:17.939845 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.939828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Aug 04 18:01:17.939845 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.939840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 18:01:17.940201 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:17.940186    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Aug 04 18:01:18.073982 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.073959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:18.081940 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.081925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-serving-cert\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.081997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.081948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-config\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.081997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.081967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-default-certificate\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.081997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.081988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlkm\" (UniqueName: \"kubernetes.io/projected/d6a9867b-b02f-4f60-8a12-783faccd03cd-kube-api-access-4wlkm\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.082092 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.082092 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4g25\" (UniqueName: \"kubernetes.io/projected/8ef40055-92c5-45a1-ad48-9f086f1292b2-kube-api-access-j4g25\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.082092 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a9867b-b02f-4f60-8a12-783faccd03cd-serving-cert\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.082182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-trusted-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.082182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989qn\" (UniqueName: \"kubernetes.io/projected/c61d8d4e-6d24-4d27-88d3-bb909065ede6-kube-api-access-989qn\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.082182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-service-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.082269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-snapshots\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.082269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-trusted-ca\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.082269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.082358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb25d\" (UniqueName: \"kubernetes.io/projected/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-kube-api-access-qb25d\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.082358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ef40055-92c5-45a1-ad48-9f086f1292b2-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.082358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-tmp\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.082447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-stats-auth\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.082447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.082394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.158249 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.158229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:18.183184 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-stats-auth\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.183184 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.183326 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-serving-cert\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.183326 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-config\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.183326 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.183302    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:18.183474 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-default-certificate\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.183474 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.183395    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.683376378 +0000 UTC m=+50.154663649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:18.183474 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlkm\" (UniqueName: \"kubernetes.io/projected/d6a9867b-b02f-4f60-8a12-783faccd03cd-kube-api-access-4wlkm\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.183667 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.183667 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4g25\" (UniqueName: \"kubernetes.io/projected/8ef40055-92c5-45a1-ad48-9f086f1292b2-kube-api-access-j4g25\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.183667 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.183631    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:18.183818 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.183685    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.683669667 +0000 UTC m=+50.154956938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:18.183818 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a9867b-b02f-4f60-8a12-783faccd03cd-serving-cert\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.183818 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-trusted-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.183818 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989qn\" (UniqueName: \"kubernetes.io/projected/c61d8d4e-6d24-4d27-88d3-bb909065ede6-kube-api-access-989qn\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.184035 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-config\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.184035 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.183940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-service-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184035 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-snapshots\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184035 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-trusted-ca\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.184230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.184230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb25d\" (UniqueName: \"kubernetes.io/projected/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-kube-api-access-qb25d\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ef40055-92c5-45a1-ad48-9f086f1292b2-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.184230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-tmp\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184434 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-service-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184492 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-tmp\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184667 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-trusted-ca-bundle\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.184736 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.184692    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.684675809 +0000 UTC m=+50.155963076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:18.184869 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.184848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d6a9867b-b02f-4f60-8a12-783faccd03cd-trusted-ca\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.185106 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.185088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-snapshots\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.185195 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.185152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8ef40055-92c5-45a1-ad48-9f086f1292b2-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.187604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.187582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-serving-cert\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.187781 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.187763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6a9867b-b02f-4f60-8a12-783faccd03cd-serving-cert\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.187781 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.187770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-default-certificate\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.187891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.187799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-stats-auth\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.192416 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.192394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlkm\" (UniqueName: \"kubernetes.io/projected/d6a9867b-b02f-4f60-8a12-783faccd03cd-kube-api-access-4wlkm\") pod \"console-operator-5ccd6b445f-pglq2\" (UID: \"d6a9867b-b02f-4f60-8a12-783faccd03cd\") " pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.192969 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.192943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4g25\" (UniqueName: \"kubernetes.io/projected/8ef40055-92c5-45a1-ad48-9f086f1292b2-kube-api-access-j4g25\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.194096 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.194073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-989qn\" (UniqueName: \"kubernetes.io/projected/c61d8d4e-6d24-4d27-88d3-bb909065ede6-kube-api-access-989qn\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.194481 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.194463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb25d\" (UniqueName: \"kubernetes.io/projected/693fec9c-5ef1-47ed-97d5-52b6d6c8b16b-kube-api-access-qb25d\") pod \"insights-operator-77696685b-wwdm7\" (UID: \"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b\") " pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.212531 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.212514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wwdm7"
Aug 04 18:01:18.224924 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.224895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:18.285465 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.285435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:18.285654 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.285532    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:18.285654 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.285607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:18.285654 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.285618    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.285597294 +0000 UTC m=+65.756884559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:18.285855 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.285769    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:18.285855 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.285790    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:18.285855 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.285844    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:01:34.285823241 +0000 UTC m=+65.757110512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:18.372679 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.372652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wwdm7"]
Aug 04 18:01:18.376284 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:18.376259    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693fec9c_5ef1_47ed_97d5_52b6d6c8b16b.slice/crio-de98d6274755d90205f48b41594b3d8b086a0b473defa714ce115f5bf2a4ae92 WatchSource:0}: Error finding container de98d6274755d90205f48b41594b3d8b086a0b473defa714ce115f5bf2a4ae92: Status 404 returned error can't find the container with id de98d6274755d90205f48b41594b3d8b086a0b473defa714ce115f5bf2a4ae92
Aug 04 18:01:18.387873 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.387853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-pglq2"]
Aug 04 18:01:18.390219 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:18.390201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a9867b_b02f_4f60_8a12_783faccd03cd.slice/crio-2c4989c52ba2c0edabd84198426821345a91987a8f540f8a70ca914710fadb3c WatchSource:0}: Error finding container 2c4989c52ba2c0edabd84198426821345a91987a8f540f8a70ca914710fadb3c: Status 404 returned error can't find the container with id 2c4989c52ba2c0edabd84198426821345a91987a8f540f8a70ca914710fadb3c
Aug 04 18:01:18.688780 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.688745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:18.688972 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.688821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.688972 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.688861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:18.688972 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.688917    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:18.689141 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.688987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:19.688968048 +0000 UTC m=+51.160255320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:18.689141 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.689005    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:18.689141 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.689007    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:19.688996768 +0000 UTC m=+51.160284020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:18.689141 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:18.689082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:19.689059555 +0000 UTC m=+51.160346822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:18.949218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:18.949125    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.011350 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.011301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:19.219325 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.219235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" event={"ID":"d6a9867b-b02f-4f60-8a12-783faccd03cd","Type":"ContainerStarted","Data":"2c4989c52ba2c0edabd84198426821345a91987a8f540f8a70ca914710fadb3c"}
Aug 04 18:01:19.220430 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.220402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wwdm7" event={"ID":"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b","Type":"ContainerStarted","Data":"de98d6274755d90205f48b41594b3d8b086a0b473defa714ce115f5bf2a4ae92"}
Aug 04 18:01:19.696451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.696411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:19.696653 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.696466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:19.696653 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:19.696568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:19.696653 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:19.696599    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:19.696653 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:19.696613    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:19.696831 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:19.696683    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:21.696661769 +0000 UTC m=+53.167949027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:19.696831 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:19.696699    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:21.69669265 +0000 UTC m=+53.167979902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:19.696831 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:19.696713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:21.696705762 +0000 UTC m=+53.167993017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:20.371059 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:20.371017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.474083 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:20.474044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.015675 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.015636    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"]
Aug 04 18:01:21.032419 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.032384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"]
Aug 04 18:01:21.032619 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.032533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.035530 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.035309    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 04 18:01:21.035530 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.035352    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 04 18:01:21.036318 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.035868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7bzk2\""
Aug 04 18:01:21.108148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.108107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/443df73d-2731-4d1c-8825-22a0b24b0a9c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.108148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.108161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.209301 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.209271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/443df73d-2731-4d1c-8825-22a0b24b0a9c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.209301 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.209302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.209527 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.209491    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:21.209583 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.209573    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:21.709558274 +0000 UTC m=+53.180845530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:21.209888 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.209872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/443df73d-2731-4d1c-8825-22a0b24b0a9c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.219569 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.219547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.388886 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.388850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.490429 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.490388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.592144 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.592115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.714223 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.714195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:21.714332 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.714243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:21.714374 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714360    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:21.714437 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.714361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:21.714437 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714426    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:25.714405851 +0000 UTC m=+57.185693117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:21.714536 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714428    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:21.714536 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714458    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:21.714536 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:22.714468394 +0000 UTC m=+54.185755650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:21.714536 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714512    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:25.714491478 +0000 UTC m=+57.185778730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:21.714675 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:21.714556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:21.714675 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:21.714655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:25.714646003 +0000 UTC m=+57.185933256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:22.229220 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.229194    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/0.log"
Aug 04 18:01:22.229381 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.229234    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6a9867b-b02f-4f60-8a12-783faccd03cd" containerID="ae84386f0c606fe7d61008d596cbe3fdeae47cdee03859f57ccfdc9239268c9d" exitCode=255
Aug 04 18:01:22.229381 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.229308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" event={"ID":"d6a9867b-b02f-4f60-8a12-783faccd03cd","Type":"ContainerDied","Data":"ae84386f0c606fe7d61008d596cbe3fdeae47cdee03859f57ccfdc9239268c9d"}
Aug 04 18:01:22.229625 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.229600    2579 scope.go:117] "RemoveContainer" containerID="ae84386f0c606fe7d61008d596cbe3fdeae47cdee03859f57ccfdc9239268c9d"
Aug 04 18:01:22.230910 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.230889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wwdm7" event={"ID":"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b","Type":"ContainerStarted","Data":"c75d81dd806e5c3a202e696688d5ffa46579ccde89285ee3fcf137022fd5caea"}
Aug 04 18:01:22.275237 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.275193    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-wwdm7" podStartSLOduration=2.098475403 podStartE2EDuration="5.27517979s" totalImagesPullingTime="3.176704387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:22.27445121 +0000 UTC m=+53.745738514" watchObservedRunningTime="2026-08-04 18:01:22.27517979 +0000 UTC m=+53.746467063"
Aug 04 18:01:22.424161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.424111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.725563 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.725524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:22.725803 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:22.725668    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:22.725803 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:22.725733    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:24.725714488 +0000 UTC m=+56.197001741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:22.735242 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.735192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.973285 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.973250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vj6wg"]
Aug 04 18:01:22.975702 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.975640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:22.979699 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:22.979673    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:01:23.008975 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.008954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:01:23.009234 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.008977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mcf9m\""
Aug 04 18:01:23.010212 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.010192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.027625 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.027605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59658784-188e-4a4e-aa98-1e9089818eee-data-volume\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.027729 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.027712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.027780 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.027739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhswx\" (UniqueName: \"kubernetes.io/projected/59658784-188e-4a4e-aa98-1e9089818eee-kube-api-access-zhswx\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.027839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.027819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59658784-188e-4a4e-aa98-1e9089818eee-crio-socket\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.027935 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.027916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.030753 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.030733    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vj6wg"]
Aug 04 18:01:23.099003 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.098975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.129258 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59658784-188e-4a4e-aa98-1e9089818eee-crio-socket\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129359 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129359 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59658784-188e-4a4e-aa98-1e9089818eee-data-volume\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59658784-188e-4a4e-aa98-1e9089818eee-crio-socket\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129492 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswx\" (UniqueName: \"kubernetes.io/projected/59658784-188e-4a4e-aa98-1e9089818eee-kube-api-access-zhswx\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.129492 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.129458    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:23.129587 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.129552    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:23.629532648 +0000 UTC m=+55.100819911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:23.129587 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.129554    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:23.129666 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.129604    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:23.629588333 +0000 UTC m=+55.100875590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:23.129732 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.129714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59658784-188e-4a4e-aa98-1e9089818eee-data-volume\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.139545 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.139525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhswx\" (UniqueName: \"kubernetes.io/projected/59658784-188e-4a4e-aa98-1e9089818eee-kube-api-access-zhswx\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.234360 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.234312    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:01:23.234752 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.234735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/0.log"
Aug 04 18:01:23.234822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.234772    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6a9867b-b02f-4f60-8a12-783faccd03cd" containerID="d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f" exitCode=255
Aug 04 18:01:23.234822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.234796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" event={"ID":"d6a9867b-b02f-4f60-8a12-783faccd03cd","Type":"ContainerDied","Data":"d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f"}
Aug 04 18:01:23.234903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.234849    2579 scope.go:117] "RemoveContainer" containerID="ae84386f0c606fe7d61008d596cbe3fdeae47cdee03859f57ccfdc9239268c9d"
Aug 04 18:01:23.235053 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.235034    2579 scope.go:117] "RemoveContainer" containerID="d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f"
Aug 04 18:01:23.235232 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.235215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-pglq2_openshift-console-operator(d6a9867b-b02f-4f60-8a12-783faccd03cd)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" podUID="d6a9867b-b02f-4f60-8a12-783faccd03cd"
Aug 04 18:01:23.633281 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.633249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.633660 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:23.633368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:23.633660 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.633419    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:23.633660 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.633485    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:24.633470681 +0000 UTC m=+56.104757933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:23.633660 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.633511    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:23.633660 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:23.633577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:24.633564203 +0000 UTC m=+56.104851456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:24.238512 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.238474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:01:24.238795 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.238780    2579 scope.go:117] "RemoveContainer" containerID="d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f"
Aug 04 18:01:24.238949 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.238931    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-pglq2_openshift-console-operator(d6a9867b-b02f-4f60-8a12-783faccd03cd)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" podUID="d6a9867b-b02f-4f60-8a12-783faccd03cd"
Aug 04 18:01:24.279650 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.279629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8vf5_21136c2d-8c4a-41d9-9b45-9d55c3599539/dns-node-resolver/0.log"
Aug 04 18:01:24.359477 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.359447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.641545 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.641489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:24.641964 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.641608    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:24.641964 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.641637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:24.641964 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.641676    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:26.641661371 +0000 UTC m=+58.112948622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:24.641964 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.641756    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:24.641964 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.641815    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:26.64179936 +0000 UTC m=+58.113086630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:24.742405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:24.742374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:24.742565 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.742533    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:24.742612 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:24.742584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:28.742572445 +0000 UTC m=+60.213859698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:25.125047 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.125017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:25.214731 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.214703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:25.279760 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.279735    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d8j6c_5082e209-4074-4742-9da7-0ce1bb74779f/node-ca/0.log"
Aug 04 18:01:25.750530 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.750463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.750592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:25.750630    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:25.750642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:25.750710    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:33.75069233 +0000 UTC m=+65.221979597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:25.750724    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:25.750745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:33.750731219 +0000 UTC m=+65.222018476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:25.750914 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:25.750766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:33.75075847 +0000 UTC m=+65.222045722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:26.091288 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.091214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6dvr_d39c81e0-45c6-4b3e-beb7-d9795902a1e8/serve-healthcheck-canary/0.log"
Aug 04 18:01:26.145733 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.145704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.243637 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.243610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.658883 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.658849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:26.659074 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.658932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:26.659074 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:26.658979    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:26.659074 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:26.659021    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:26.659074 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:26.659056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.659037228 +0000 UTC m=+62.130324498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:26.659237 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:26.659083    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.659065845 +0000 UTC m=+62.130353097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:26.755818 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.755766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:26.857208 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:26.857176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.156608 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:27.156582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.257306 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:27.257272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.357171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:27.357136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.457289 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:27.457215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.559798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:27.559762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:28.225197 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:28.225154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:28.225727 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:28.225225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:28.225727 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:28.225661    2579 scope.go:117] "RemoveContainer" containerID="d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f"
Aug 04 18:01:28.225879 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:28.225854    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-pglq2_openshift-console-operator(d6a9867b-b02f-4f60-8a12-783faccd03cd)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" podUID="d6a9867b-b02f-4f60-8a12-783faccd03cd"
Aug 04 18:01:28.775584 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:28.775542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:28.775740 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:28.775697    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:28.775797 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:28.775767    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:36.775750534 +0000 UTC m=+68.247037786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:29.105868 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:29.105795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.168675 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:29.168649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dwcm5"
Aug 04 18:01:29.328230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:29.328192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.393915 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:29.393873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.495426 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:29.495394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:30.692829 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:30.692791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:30.693242 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:30.692880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:30.693242 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:30.692913    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:30.693242 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:30.692984    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:30.693242 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:30.692990    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:38.692974779 +0000 UTC m=+70.164262030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:30.693242 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:30.693022    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:38.693010981 +0000 UTC m=+70.164298233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:31.222657 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:31.222614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:31.459551 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:31.459520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:33.801315 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:33.801276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:33.818451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:33.818429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:33.818534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:33.818468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:33.818574 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:33.818556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:33.818607 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:33.818589    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:33.818660 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:33.818649    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:49.818633134 +0000 UTC m=+81.289920386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : secret "router-metrics-certs-default" not found
Aug 04 18:01:33.818703 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:33.818649    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:33.818703 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:33.818688    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle podName:c61d8d4e-6d24-4d27-88d3-bb909065ede6 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:49.818675296 +0000 UTC m=+81.289962556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle") pod "router-default-7f6d8894dd-p9sxx" (UID: "c61d8d4e-6d24-4d27-88d3-bb909065ede6") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:33.818766 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:33.818705    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls podName:8ef40055-92c5-45a1-ad48-9f086f1292b2 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:49.818698216 +0000 UTC m=+81.289985468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-n562v" (UID: "8ef40055-92c5-45a1-ad48-9f086f1292b2") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:01:33.900450 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:33.900424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.002037 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.002011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.101907 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.101844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.322750 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.322731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:01:34.322868 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.322801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:01:34.322930 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.322884    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:34.322930 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.322910    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:01:34.322930 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.322919    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57545cc597-5c2b4: secret "image-registry-tls" not found
Aug 04 18:01:34.323054 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.322947    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls podName:008642e1-a70d-496a-876c-e70fc8b02be0 nodeName:}" failed. No retries permitted until 2026-08-04 18:02:06.322929601 +0000 UTC m=+97.794216853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls") pod "dns-default-bbmqg" (UID: "008642e1-a70d-496a-876c-e70fc8b02be0") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:34.323054 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.322968    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls podName:c3cb065b-280b-4d47-b5db-7c0cdfbd43fe nodeName:}" failed. No retries permitted until 2026-08-04 18:02:06.32295682 +0000 UTC m=+97.794244072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls") pod "image-registry-57545cc597-5c2b4" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe") : secret "image-registry-tls" not found
Aug 04 18:01:34.725581 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.725555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:34.728191 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.728171    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:34.738487 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.738463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d06c764-6c99-4554-a19c-b8f08450baef-original-pull-secret\") pod \"global-pull-secret-syncer-m5ldx\" (UID: \"7d06c764-6c99-4554-a19c-b8f08450baef\") " pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:34.750655 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.750629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:34.825814 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.825788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:01:34.826085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.825840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:34.828591 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.828578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:34.828653 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.828604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:34.836991 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.836970    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Aug 04 18:01:34.837047 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:34.837023    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs podName:2550b4e7-088e-48e4-abc9-2a500311fabc nodeName:}" failed. No retries permitted until 2026-08-04 18:02:38.837009169 +0000 UTC m=+130.308296422 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs") pod "network-metrics-daemon-jw5pt" (UID: "2550b4e7-088e-48e4-abc9-2a500311fabc") : secret "metrics-daemon-secret" not found
Aug 04 18:01:34.842426 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.842410    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:34.849125 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.849110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq77\" (UniqueName: \"kubernetes.io/projected/56060bb1-3104-4bbb-beb9-008c28976663-kube-api-access-rrq77\") pod \"network-check-target-tglfd\" (UID: \"56060bb1-3104-4bbb-beb9-008c28976663\") " pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:34.946056 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.946038    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x566h\""
Aug 04 18:01:34.949279 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.949265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5ldx"
Aug 04 18:01:34.953393 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:34.953378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:35.094300 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.094269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5ldx"]
Aug 04 18:01:35.099383 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:35.099355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d06c764_6c99_4554_a19c_b8f08450baef.slice/crio-662f6917db151f7a3e97a0bf5e8564e00a58241a63f90c685940ce18c02eab0b WatchSource:0}: Error finding container 662f6917db151f7a3e97a0bf5e8564e00a58241a63f90c685940ce18c02eab0b: Status 404 returned error can't find the container with id 662f6917db151f7a3e97a0bf5e8564e00a58241a63f90c685940ce18c02eab0b
Aug 04 18:01:35.111664 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.111642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tglfd"]
Aug 04 18:01:35.114710 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:35.114685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56060bb1_3104_4bbb_beb9_008c28976663.slice/crio-ecfeebb2da34e34b9ad5fa724c652c977439d27885b622f0b8234c84338f22ac WatchSource:0}: Error finding container ecfeebb2da34e34b9ad5fa724c652c977439d27885b622f0b8234c84338f22ac: Status 404 returned error can't find the container with id ecfeebb2da34e34b9ad5fa724c652c977439d27885b622f0b8234c84338f22ac
Aug 04 18:01:35.264689 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.264621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tglfd" event={"ID":"56060bb1-3104-4bbb-beb9-008c28976663","Type":"ContainerStarted","Data":"ecfeebb2da34e34b9ad5fa724c652c977439d27885b622f0b8234c84338f22ac"}
Aug 04 18:01:35.265670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.265652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5ldx" event={"ID":"7d06c764-6c99-4554-a19c-b8f08450baef","Type":"ContainerStarted","Data":"662f6917db151f7a3e97a0bf5e8564e00a58241a63f90c685940ce18c02eab0b"}
Aug 04 18:01:35.766448 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.766381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:35.868002 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:35.867974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.356098 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.356060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.425735 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.425677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.757802 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.757754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.816628 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.816593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.845076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.845045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:36.845226 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:36.845200    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:01:36.845343 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:36.845272    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert podName:443df73d-2731-4d1c-8825-22a0b24b0a9c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:52.845254524 +0000 UTC m=+84.316541776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-zdzfm" (UID: "443df73d-2731-4d1c-8825-22a0b24b0a9c") : secret "networking-console-plugin-cert" not found
Aug 04 18:01:36.950781 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:36.950756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:37.152817 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:37.152781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:37.355254 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:37.355221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:37.514276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:37.514191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:37.938378 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:37.938342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.311114 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.311027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.410329 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.410296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.549524 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.549446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:38.760745 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.760710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:38.760938 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.760823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:38.761005 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:38.760944    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:38.761061 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:01:38.761020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls podName:59658784-188e-4a4e-aa98-1e9089818eee nodeName:}" failed. No retries permitted until 2026-08-04 18:01:54.761000034 +0000 UTC m=+86.232287290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vj6wg" (UID: "59658784-188e-4a4e-aa98-1e9089818eee") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:38.761368 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.761334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59658784-188e-4a4e-aa98-1e9089818eee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:38.953665 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:38.953626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.158381 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:39.158343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.582999 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:39.582968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.280712 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.280675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tglfd" event={"ID":"56060bb1-3104-4bbb-beb9-008c28976663","Type":"ContainerStarted","Data":"d4b3a418a4656139ad7c01246f2bce1af8e5266bdd8012869b8241c8f10f446b"}
Aug 04 18:01:40.280904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.280782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:01:40.282047 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.282028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5ldx" event={"ID":"7d06c764-6c99-4554-a19c-b8f08450baef","Type":"ContainerStarted","Data":"83cdde82730a1ec7c7d03d257636a79a416a6855f9f0b550c1cc684839258420"}
Aug 04 18:01:40.302549 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.302489    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tglfd" podStartSLOduration=66.930229164 podStartE2EDuration="1m11.302476228s" totalImagesPullingTime="4.372247064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:40.301456554 +0000 UTC m=+71.772743829" watchObservedRunningTime="2026-08-04 18:01:40.302476228 +0000 UTC m=+71.773763501"
Aug 04 18:01:40.322814 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.322774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m5ldx" podStartSLOduration=66.931074846 podStartE2EDuration="1m11.322762798s" totalImagesPullingTime="4.391687952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:40.322003645 +0000 UTC m=+71.793290919" watchObservedRunningTime="2026-08-04 18:01:40.322762798 +0000 UTC m=+71.794050072"
Aug 04 18:01:40.351032 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.351011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.550858 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.550778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.950907 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:40.950878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:41.170141 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:41.170087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:41.552567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:41.552532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:41.948190 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:41.948155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:42.158310 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:42.158279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:43.025218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.025187    2579 scope.go:117] "RemoveContainer" containerID="d0c87bec24a66a6a5d7018b4ea7adc1bab30b9a26b66ac175fb1c651c26d8a6f"
Aug 04 18:01:43.291538 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.291447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:01:43.291905 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.291553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" event={"ID":"d6a9867b-b02f-4f60-8a12-783faccd03cd","Type":"ContainerStarted","Data":"6fe93eb3bec0b0bbf198da4e31c8982b8b17f1e2e8dad91a7f0535bd189f161b"}
Aug 04 18:01:43.291905 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.291815    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:43.309764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.309708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2" podStartSLOduration=23.15134211 podStartE2EDuration="26.309663932s" totalImagesPullingTime="3.158321822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:43.308697515 +0000 UTC m=+74.779984792" watchObservedRunningTime="2026-08-04 18:01:43.309663932 +0000 UTC m=+74.780951207"
Aug 04 18:01:43.618392 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.618365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-pglq2"
Aug 04 18:01:43.794288 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.794259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:43.894108 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.894031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:43.955509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:43.955464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.110328 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:44.110299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.475888 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:44.475832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.575926 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:44.575891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.905911 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:44.905878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.963450 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:44.963417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:47.994301 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:47.994260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:48.050626 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:48.050596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:49.849636 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.849602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:49.849999 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.849645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:49.849999 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.849725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:49.850357 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.850328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c61d8d4e-6d24-4d27-88d3-bb909065ede6-service-ca-bundle\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:49.852080 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.852053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8ef40055-92c5-45a1-ad48-9f086f1292b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-n562v\" (UID: \"8ef40055-92c5-45a1-ad48-9f086f1292b2\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:49.852176 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.852061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61d8d4e-6d24-4d27-88d3-bb909065ede6-metrics-certs\") pod \"router-default-7f6d8894dd-p9sxx\" (UID: \"c61d8d4e-6d24-4d27-88d3-bb909065ede6\") " pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:49.995399 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:49.995372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hrp4p\""
Aug 04 18:01:50.003567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.003554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"
Aug 04 18:01:50.046267 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.046242    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-h78mq\""
Aug 04 18:01:50.053983 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.053959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:50.131556 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.131529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v"]
Aug 04 18:01:50.134001 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:50.133976    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef40055_92c5_45a1_ad48_9f086f1292b2.slice/crio-62cdb225a9149a4d65eb5c124bcbe585568bc51492e4034c5363738ce0daf116 WatchSource:0}: Error finding container 62cdb225a9149a4d65eb5c124bcbe585568bc51492e4034c5363738ce0daf116: Status 404 returned error can't find the container with id 62cdb225a9149a4d65eb5c124bcbe585568bc51492e4034c5363738ce0daf116
Aug 04 18:01:50.201383 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.199574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f6d8894dd-p9sxx"]
Aug 04 18:01:50.202792 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:50.202765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61d8d4e_6d24_4d27_88d3_bb909065ede6.slice/crio-a6c1e35ae5ce0a349aad758be143725fcb98539b655411d983386bb15d0479c4 WatchSource:0}: Error finding container a6c1e35ae5ce0a349aad758be143725fcb98539b655411d983386bb15d0479c4: Status 404 returned error can't find the container with id a6c1e35ae5ce0a349aad758be143725fcb98539b655411d983386bb15d0479c4
Aug 04 18:01:50.315287 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.315254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx" event={"ID":"c61d8d4e-6d24-4d27-88d3-bb909065ede6","Type":"ContainerStarted","Data":"db386b70cf88e2b3f67b5c534cb209d6c340fce15a40478f271ac14b5b16a495"}
Aug 04 18:01:50.315437 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.315302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx" event={"ID":"c61d8d4e-6d24-4d27-88d3-bb909065ede6","Type":"ContainerStarted","Data":"a6c1e35ae5ce0a349aad758be143725fcb98539b655411d983386bb15d0479c4"}
Aug 04 18:01:50.316365 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:50.316337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v" event={"ID":"8ef40055-92c5-45a1-ad48-9f086f1292b2","Type":"ContainerStarted","Data":"62cdb225a9149a4d65eb5c124bcbe585568bc51492e4034c5363738ce0daf116"}
Aug 04 18:01:51.054952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.054918    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:51.058000 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.057972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:51.079171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.079121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx" podStartSLOduration=34.079107626 podStartE2EDuration="34.079107626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:50.335337833 +0000 UTC m=+81.806625107" watchObservedRunningTime="2026-08-04 18:01:51.079107626 +0000 UTC m=+82.550394878"
Aug 04 18:01:51.230682 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.230650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.290233 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.290200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.322204 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.322114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:51.323534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.323411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f6d8894dd-p9sxx"
Aug 04 18:01:51.653517 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.653466    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"]
Aug 04 18:01:51.656805 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.656781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.660428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.660356    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 04 18:01:51.660634 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.660613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 18:01:51.661072 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.661049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 18:01:51.661668 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.661491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 04 18:01:51.661668 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.661523    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 18:01:51.661668 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.661523    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 04 18:01:51.661964 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.661945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 04 18:01:51.671717 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.671691    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"]
Aug 04 18:01:51.701765 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.701732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.767926 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.767905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0bbf8-9896-4256-8cbf-42d9198a8513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.768015 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.767960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.768015 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.768002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.768104 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.768061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.768143 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.768121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/0dd0bbf8-9896-4256-8cbf-42d9198a8513-kube-api-access-bst47\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.768181 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.768140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.799741 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.799716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.869192 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0bbf8-9896-4256-8cbf-42d9198a8513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869367 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869367 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869367 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869559 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/0dd0bbf8-9896-4256-8cbf-42d9198a8513-kube-api-access-bst47\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869559 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.869993 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.869970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd0bbf8-9896-4256-8cbf-42d9198a8513-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.873869 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.873846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-ca\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.874004 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.873964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.874103 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.874076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.874234 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.874153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dd0bbf8-9896-4256-8cbf-42d9198a8513-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.874833 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.874808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.881646 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.881624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst47\" (UniqueName: \"kubernetes.io/projected/0dd0bbf8-9896-4256-8cbf-42d9198a8513-kube-api-access-bst47\") pod \"cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t\" (UID: \"0dd0bbf8-9896-4256-8cbf-42d9198a8513\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:51.974354 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.974296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:51.979299 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:51.979279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"
Aug 04 18:01:52.071147 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.071014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:52.107050 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.107031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t"]
Aug 04 18:01:52.109927 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:52.109906    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dd0bbf8_9896_4256_8cbf_42d9198a8513.slice/crio-5899624b8d3787ef18f9e7d4b114df2dc99dad4fe0558055f442f9261f1d3c35 WatchSource:0}: Error finding container 5899624b8d3787ef18f9e7d4b114df2dc99dad4fe0558055f442f9261f1d3c35: Status 404 returned error can't find the container with id 5899624b8d3787ef18f9e7d4b114df2dc99dad4fe0558055f442f9261f1d3c35
Aug 04 18:01:52.133634 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.133610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:52.248100 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.248040    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:52.326659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.326634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v" event={"ID":"8ef40055-92c5-45a1-ad48-9f086f1292b2","Type":"ContainerStarted","Data":"31c01cf843382bad2ca4d21aa559cca2bb9e6960fcca9066650d0f24408a9c4b"}
Aug 04 18:01:52.327923 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.327898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t" event={"ID":"0dd0bbf8-9896-4256-8cbf-42d9198a8513","Type":"ContainerStarted","Data":"5899624b8d3787ef18f9e7d4b114df2dc99dad4fe0558055f442f9261f1d3c35"}
Aug 04 18:01:52.345973 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.345935    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-n562v" podStartSLOduration=33.752040924 podStartE2EDuration="35.345922102s" totalImagesPullingTime="1.593881178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:52.345317352 +0000 UTC m=+83.816604645" watchObservedRunningTime="2026-08-04 18:01:52.345922102 +0000 UTC m=+83.817209375"
Aug 04 18:01:52.879202 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.879166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:52.882322 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:52.882273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/443df73d-2731-4d1c-8825-22a0b24b0a9c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-zdzfm\" (UID: \"443df73d-2731-4d1c-8825-22a0b24b0a9c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:53.038915 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.038879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:53.128239 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.128206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:53.146719 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.146653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7bzk2\""
Aug 04 18:01:53.154904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.154881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"
Aug 04 18:01:53.334076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.331206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm"]
Aug 04 18:01:53.341052 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:53.341016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm" event={"ID":"443df73d-2731-4d1c-8825-22a0b24b0a9c","Type":"ContainerStarted","Data":"676ddba6497da793c6bcb8725c15dd78bd4149f00bc6e8abb05639bec6f48884"}
Aug 04 18:01:54.799863 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:54.799822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:54.802839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:54.802813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59658784-188e-4a4e-aa98-1e9089818eee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vj6wg\" (UID: \"59658784-188e-4a4e-aa98-1e9089818eee\") " pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:55.088723 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:55.088644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mcf9m\""
Aug 04 18:01:55.095988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:55.095967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vj6wg"
Aug 04 18:01:55.629843 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:55.629819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vj6wg"]
Aug 04 18:01:55.637540 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:01:55.637515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59658784_188e_4a4e_aa98_1e9089818eee.slice/crio-be426fedb7523106955f16dc75f814a223eb82270c98600ae68519250c40e6dc WatchSource:0}: Error finding container be426fedb7523106955f16dc75f814a223eb82270c98600ae68519250c40e6dc: Status 404 returned error can't find the container with id be426fedb7523106955f16dc75f814a223eb82270c98600ae68519250c40e6dc
Aug 04 18:01:56.353598 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.353571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm" event={"ID":"443df73d-2731-4d1c-8825-22a0b24b0a9c","Type":"ContainerStarted","Data":"25bbca84a83f75092c9f8111196c169dcc679706ba5f3f3e8cf49a9c64eaae98"}
Aug 04 18:01:56.355029 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.355008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj6wg" event={"ID":"59658784-188e-4a4e-aa98-1e9089818eee","Type":"ContainerStarted","Data":"e0f101fbc10a55d4a1474d7676565c4d351ed5a477fde7c5013f299f7d56ef7e"}
Aug 04 18:01:56.355133 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.355060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj6wg" event={"ID":"59658784-188e-4a4e-aa98-1e9089818eee","Type":"ContainerStarted","Data":"be426fedb7523106955f16dc75f814a223eb82270c98600ae68519250c40e6dc"}
Aug 04 18:01:56.356834 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.356814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t" event={"ID":"0dd0bbf8-9896-4256-8cbf-42d9198a8513","Type":"ContainerStarted","Data":"9b5c4013c3834b60e294bc6bd3063e1fa45518257957fc4f68bb42b976607fe1"}
Aug 04 18:01:56.356919 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.356848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t" event={"ID":"0dd0bbf8-9896-4256-8cbf-42d9198a8513","Type":"ContainerStarted","Data":"b997c0695f64cac03e6aa2de5c15ce9be2babda2e95bcd31c76b6dad248cb71f"}
Aug 04 18:01:56.356919 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.356865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t" event={"ID":"0dd0bbf8-9896-4256-8cbf-42d9198a8513","Type":"ContainerStarted","Data":"9e1c0b461aa4a78770b5e0feed536cc2ede2c684696f42516bd69556fc1c37f5"}
Aug 04 18:01:56.372190 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.372142    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-zdzfm" podStartSLOduration=33.511327485 podStartE2EDuration="36.372123657s" totalImagesPullingTime="2.860796172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:56.370934031 +0000 UTC m=+87.842221330" watchObservedRunningTime="2026-08-04 18:01:56.372123657 +0000 UTC m=+87.843410932"
Aug 04 18:01:56.394647 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.394610    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f7bc8bd4b-7sc5t" podStartSLOduration=1.970558349 podStartE2EDuration="5.394597335s" totalImagesPullingTime="3.424038986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:56.39411179 +0000 UTC m=+87.865399065" watchObservedRunningTime="2026-08-04 18:01:56.394597335 +0000 UTC m=+87.865884608"
Aug 04 18:01:56.500303 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.500218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:56.599857 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:56.599822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:57.362693 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:57.362652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj6wg" event={"ID":"59658784-188e-4a4e-aa98-1e9089818eee","Type":"ContainerStarted","Data":"04abd5b6d7c67791af3e534e8549bb7380f33e2ae4c082c3e135575ecaddcf3b"}
Aug 04 18:01:58.367927 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:58.367893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vj6wg" event={"ID":"59658784-188e-4a4e-aa98-1e9089818eee","Type":"ContainerStarted","Data":"8bddfbbf3ad835703c00cb248721c8de6f1e40380f5e247ebed39ce9511016e0"}
Aug 04 18:01:58.388558 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:58.388481    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vj6wg" podStartSLOduration=34.23205235 podStartE2EDuration="36.388463701s" totalImagesPullingTime="2.156411351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:58.387311255 +0000 UTC m=+89.858598550" watchObservedRunningTime="2026-08-04 18:01:58.388463701 +0000 UTC m=+89.859750975"
Aug 04 18:01:58.445632 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:58.445598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:58.546783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:58.546752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:58.602112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:01:58.602082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:00.274170 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:00.274137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:00.353343 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:00.353309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:00.639760 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:00.639728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.113358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:01.113280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.169865 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:01.169839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.565983 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:01.565956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.628046 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:01.628015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:02.766929 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:02.766891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:04.518961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:04.518932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:04.573943 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:04.573909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:05.406274 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.406245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:05.634768 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.634740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:05.686939 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.686858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-47pcn"]
Aug 04 18:02:05.689580 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.689565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.692513 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.692485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:02:05.692611 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.692555    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:02:05.693548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.693530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:02:05.693628 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.693532    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:02:05.693628 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.693584    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zlmsf\""
Aug 04 18:02:05.724271 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.724249    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:05.784366 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.784343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:05.789509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnj55\" (UniqueName: \"kubernetes.io/projected/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-kube-api-access-lnj55\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-textfile\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789672 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789672 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-root\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789743 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789797 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-wtmp\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789853 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-metrics-client-ca\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.789853 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.789834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-sys\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891158 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891403 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnj55\" (UniqueName: \"kubernetes.io/projected/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-kube-api-access-lnj55\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891532 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-textfile\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891532 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891649 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-root\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891649 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891649 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-wtmp\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-metrics-client-ca\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-sys\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891886 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-textfile\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891886 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-root\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.891886 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:02:05.891835    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Aug 04 18:02:05.891886 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-sys\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.892076 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:02:05.891910    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls podName:4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d nodeName:}" failed. No retries permitted until 2026-08-04 18:02:06.391890584 +0000 UTC m=+97.863177852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls") pod "node-exporter-47pcn" (UID: "4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d") : secret "node-exporter-tls" not found
Aug 04 18:02:05.892076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.891952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-wtmp\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.892525 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.892488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-metrics-client-ca\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.892638 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.892484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-accelerators-collector-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.893912 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.893894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.903218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.903188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnj55\" (UniqueName: \"kubernetes.io/projected/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-kube-api-access-lnj55\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:05.915225 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:05.915202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.015091 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.015027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.080897 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.080872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.193156 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.193123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.394933 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.394905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:06.395076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.394954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:06.395076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.394996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:02:06.397365 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.397338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d-node-exporter-tls\") pod \"node-exporter-47pcn\" (UID: \"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d\") " pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:06.397482 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.397399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"image-registry-57545cc597-5c2b4\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") " pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:06.397482 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.397465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/008642e1-a70d-496a-876c-e70fc8b02be0-metrics-tls\") pod \"dns-default-bbmqg\" (UID: \"008642e1-a70d-496a-876c-e70fc8b02be0\") " pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:02:06.598727 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.598697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-47pcn"
Aug 04 18:02:06.611513 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:02:06.611453    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcf0a2e_be82_4ed5_89d8_2665f0ccca8d.slice/crio-855ba5f78ef1bc6366d1f0f9e4f529192394ff21664596976059da8aa8d9ae3e WatchSource:0}: Error finding container 855ba5f78ef1bc6366d1f0f9e4f529192394ff21664596976059da8aa8d9ae3e: Status 404 returned error can't find the container with id 855ba5f78ef1bc6366d1f0f9e4f529192394ff21664596976059da8aa8d9ae3e
Aug 04 18:02:06.621920 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.621900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9xbrz\""
Aug 04 18:02:06.629891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.629870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:06.637536 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.637518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nwtj6\""
Aug 04 18:02:06.645537 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.645477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:02:06.650274 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.650251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.748440 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.748356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.777623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.777593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:02:06.780866 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:02:06.780843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cb065b_280b_4d47_b5db_7c0cdfbd43fe.slice/crio-2406bf4b5a13666b8b908aa548cca38d2b9a46a9aee415908777bd8180307907 WatchSource:0}: Error finding container 2406bf4b5a13666b8b908aa548cca38d2b9a46a9aee415908777bd8180307907: Status 404 returned error can't find the container with id 2406bf4b5a13666b8b908aa548cca38d2b9a46a9aee415908777bd8180307907
Aug 04 18:02:06.788462 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.788440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bbmqg"]
Aug 04 18:02:06.792363 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:02:06.792341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008642e1_a70d_496a_876c_e70fc8b02be0.slice/crio-455de2384f2bf3065332ce0dbe5bc24e849a3dd8c5678aa8142e083c2ccbe8d9 WatchSource:0}: Error finding container 455de2384f2bf3065332ce0dbe5bc24e849a3dd8c5678aa8142e083c2ccbe8d9: Status 404 returned error can't find the container with id 455de2384f2bf3065332ce0dbe5bc24e849a3dd8c5678aa8142e083c2ccbe8d9
Aug 04 18:02:06.963195 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:06.963112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:07.397038 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.396999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" event={"ID":"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe","Type":"ContainerStarted","Data":"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"}
Aug 04 18:02:07.397212 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.397060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" event={"ID":"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe","Type":"ContainerStarted","Data":"2406bf4b5a13666b8b908aa548cca38d2b9a46a9aee415908777bd8180307907"}
Aug 04 18:02:07.397212 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.397110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:07.398395 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.398367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47pcn" event={"ID":"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d","Type":"ContainerStarted","Data":"855ba5f78ef1bc6366d1f0f9e4f529192394ff21664596976059da8aa8d9ae3e"}
Aug 04 18:02:07.399609 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.399587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbmqg" event={"ID":"008642e1-a70d-496a-876c-e70fc8b02be0","Type":"ContainerStarted","Data":"455de2384f2bf3065332ce0dbe5bc24e849a3dd8c5678aa8142e083c2ccbe8d9"}
Aug 04 18:02:07.430729 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:07.428347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" podStartSLOduration=98.428328974 podStartE2EDuration="1m38.428328974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:07.426601572 +0000 UTC m=+98.897888846" watchObservedRunningTime="2026-08-04 18:02:07.428328974 +0000 UTC m=+98.899616249"
Aug 04 18:02:08.110806 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.110767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.243830 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.243797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.345012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.344976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.405344 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.405303    2579 generic.go:356] "Generic (PLEG): container finished" podID="4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d" containerID="4c09ab22864c8905667c8e28e934797a096ef0db55bd3fb5003870cb71a32cce" exitCode=0
Aug 04 18:02:08.405528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.405395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47pcn" event={"ID":"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d","Type":"ContainerDied","Data":"4c09ab22864c8905667c8e28e934797a096ef0db55bd3fb5003870cb71a32cce"}
Aug 04 18:02:08.602970 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.602941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.660309 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.660233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.874146 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.874096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.974191 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:08.974104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:09.410390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:09.410362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47pcn" event={"ID":"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d","Type":"ContainerStarted","Data":"1b82d38c078066038f2fdc9e965659ff93e8947d47bbccac5dde638e50239c6f"}
Aug 04 18:02:09.410782 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:09.410419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-47pcn" event={"ID":"4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d","Type":"ContainerStarted","Data":"59c57c80def410873683f1bfd27fcfa0b6d30e2953e816ef583949e03997decb"}
Aug 04 18:02:09.432837 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:09.432792    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-47pcn" podStartSLOduration=3.425462246 podStartE2EDuration="4.432778011s" totalImagesPullingTime="943.933784ms" totalInitContainerRuntime="63.381981ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:09.431925831 +0000 UTC m=+100.903213104" watchObservedRunningTime="2026-08-04 18:02:09.432778011 +0000 UTC m=+100.904065287"
Aug 04 18:02:10.415437 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.415403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbmqg" event={"ID":"008642e1-a70d-496a-876c-e70fc8b02be0","Type":"ContainerStarted","Data":"49840a0585660cc2a97b346e93a034f4a1f30eb7256a6ea0486d0edf5dd87a78"}
Aug 04 18:02:10.415867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.415458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bbmqg" event={"ID":"008642e1-a70d-496a-876c-e70fc8b02be0","Type":"ContainerStarted","Data":"33c1f4319b955dfe0b3d2da2f0825c6e36b7eca0f700e89a76f590fad68d1903"}
Aug 04 18:02:10.415867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.415534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:02:10.445421 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.445397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:10.445998 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.445961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bbmqg" podStartSLOduration=65.666200613 podStartE2EDuration="1m8.445948437s" totalImagesPullingTime="2.779747824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:10.443231396 +0000 UTC m=+101.914518691" watchObservedRunningTime="2026-08-04 18:02:10.445948437 +0000 UTC m=+101.917235745"
Aug 04 18:02:10.532228 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:10.532191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:11.286774 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:11.286742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tglfd"
Aug 04 18:02:12.110839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:12.110811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:12.207347 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:12.207316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:14.187580 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:14.187545    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:02:14.286465 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:14.286433    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:14.382434 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:14.382407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:17.320457 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:17.320425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:17.399802 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:17.399775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:18.709003 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:18.708966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:18.769447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:18.769415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:20.421103 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:20.421067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bbmqg"
Aug 04 18:02:21.145819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:21.145787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:21.204982 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:21.204953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:22.697685 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:22.697653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:24.193809 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:24.193777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:26.772896 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:26.772863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:26.872569 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:26.872520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:26.972622 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:26.972599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:27.473875 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:27.473841    2579 generic.go:356] "Generic (PLEG): container finished" podID="693fec9c-5ef1-47ed-97d5-52b6d6c8b16b" containerID="c75d81dd806e5c3a202e696688d5ffa46579ccde89285ee3fcf137022fd5caea" exitCode=0
Aug 04 18:02:27.473989 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:27.473906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wwdm7" event={"ID":"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b","Type":"ContainerDied","Data":"c75d81dd806e5c3a202e696688d5ffa46579ccde89285ee3fcf137022fd5caea"}
Aug 04 18:02:27.474205 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:27.474190    2579 scope.go:117] "RemoveContainer" containerID="c75d81dd806e5c3a202e696688d5ffa46579ccde89285ee3fcf137022fd5caea"
Aug 04 18:02:27.801148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:27.801123    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:27.862871 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:27.862839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:28.479603 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:28.479565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wwdm7" event={"ID":"693fec9c-5ef1-47ed-97d5-52b6d6c8b16b","Type":"ContainerStarted","Data":"8b0c2bdaae07cae0702916642fa01bd9e1927ce1947ee629d7d94ce90e2d823c"}
Aug 04 18:02:29.570286 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:29.570256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:29.637840 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:29.637813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:33.943731 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:33.943687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:34.043736 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:34.043703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:38.932127 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:38.932090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:02:38.934437 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:38.934407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2550b4e7-088e-48e4-abc9-2a500311fabc-metrics-certs\") pod \"network-metrics-daemon-jw5pt\" (UID: \"2550b4e7-088e-48e4-abc9-2a500311fabc\") " pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:02:39.140112 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.140065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bbhlr\""
Aug 04 18:02:39.147609 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.147588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jw5pt"
Aug 04 18:02:39.207461 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.207304    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" podUID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" containerName="registry" containerID="cri-o://4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7" gracePeriod=30
Aug 04 18:02:39.303207 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.303173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jw5pt"]
Aug 04 18:02:39.306312 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:02:39.306284    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2550b4e7_088e_48e4_abc9_2a500311fabc.slice/crio-646d62cc39759d3b680fcc3ed5ffb0e80a0d9ae996ecdfbd66905851ce4d7dda WatchSource:0}: Error finding container 646d62cc39759d3b680fcc3ed5ffb0e80a0d9ae996ecdfbd66905851ce4d7dda: Status 404 returned error can't find the container with id 646d62cc39759d3b680fcc3ed5ffb0e80a0d9ae996ecdfbd66905851ce4d7dda
Aug 04 18:02:39.455859 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.455834    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:39.522355 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.522271    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" containerID="4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7" exitCode=0
Aug 04 18:02:39.522355 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.522310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57545cc597-5c2b4"
Aug 04 18:02:39.522596 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.522347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" event={"ID":"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe","Type":"ContainerDied","Data":"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"}
Aug 04 18:02:39.522596 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.522401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57545cc597-5c2b4" event={"ID":"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe","Type":"ContainerDied","Data":"2406bf4b5a13666b8b908aa548cca38d2b9a46a9aee415908777bd8180307907"}
Aug 04 18:02:39.522596 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.522424    2579 scope.go:117] "RemoveContainer" containerID="4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"
Aug 04 18:02:39.523521 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.523424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw5pt" event={"ID":"2550b4e7-088e-48e4-abc9-2a500311fabc","Type":"ContainerStarted","Data":"646d62cc39759d3b680fcc3ed5ffb0e80a0d9ae996ecdfbd66905851ce4d7dda"}
Aug 04 18:02:39.530604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.530566    2579 scope.go:117] "RemoveContainer" containerID="4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"
Aug 04 18:02:39.530896 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:02:39.530871    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7\": container with ID starting with 4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7 not found: ID does not exist" containerID="4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"
Aug 04 18:02:39.530992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.530902    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7"} err="failed to get container status \"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7\": rpc error: code = NotFound desc = could not find container \"4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7\": container with ID starting with 4661f531867901f85b6fb51bf768c4b20c4904e0c94326cef7a8ebf19537b4d7 not found: ID does not exist"
Aug 04 18:02:39.538123 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svwv7\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538223    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token\") pod \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\" (UID: \"c3cb065b-280b-4d47-b5db-7c0cdfbd43fe\") "
Aug 04 18:02:39.538628 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:39.538790 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.538764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:02:39.541312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.541280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:39.541417 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.541321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7" (OuterVolumeSpecName: "kube-api-access-svwv7") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "kube-api-access-svwv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:39.541417 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.541373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:02:39.541557 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.541422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:39.541557 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.541525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:02:39.548185 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.548158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" (UID: "c3cb065b-280b-4d47-b5db-7c0cdfbd43fe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:02:39.639026 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.638988    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwv7\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-kube-api-access-svwv7\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639026 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639027    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-image-registry-private-configuration\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639041    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-ca-trust-extracted\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639054    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-certificates\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639065    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-installation-pull-secrets\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639077    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-bound-sa-token\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639088    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-trusted-ca\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.639276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.639098    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe-registry-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:02:39.849373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.849343    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:02:39.856738 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:39.856711    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57545cc597-5c2b4"]
Aug 04 18:02:41.033447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.033417    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" path="/var/lib/kubelet/pods/c3cb065b-280b-4d47-b5db-7c0cdfbd43fe/volumes"
Aug 04 18:02:41.535733 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.535702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw5pt" event={"ID":"2550b4e7-088e-48e4-abc9-2a500311fabc","Type":"ContainerStarted","Data":"60d928c006b55c908d3fa09c744f5b53829395821899e686ea3433d27efeaf5d"}
Aug 04 18:02:41.535876 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.535753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jw5pt" event={"ID":"2550b4e7-088e-48e4-abc9-2a500311fabc","Type":"ContainerStarted","Data":"e2597a2c4d99a69964a56f057d271a287c68226ce195f20f75e92fdcc0b83ec7"}
Aug 04 18:02:41.555401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.555358    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jw5pt" podStartSLOduration=131.368502638 podStartE2EDuration="2m12.555342963s" totalImagesPullingTime="1.186840325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:41.553515572 +0000 UTC m=+133.024802859" watchObservedRunningTime="2026-08-04 18:02:41.555342963 +0000 UTC m=+133.026630237"
Aug 04 18:02:41.668543 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.668513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:41.765701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.765677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:41.833188 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:41.833131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:48.935783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:48.935741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:49.003171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:02:49.003122    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:21.762021 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:21.761946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:21.819534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:21.819487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:22.669195 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:22.669169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:22.735678 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:22.735649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:33.674809 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:33.674757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:33.788859 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:33.788828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:34.067898 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:34.067825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:34.169344 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:34.169312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:34.225467 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:34.225439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:44.751289 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:44.751250    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:44.850330 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:44.850303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:58.469091 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:58.469054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:58.569856 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:03:58.569826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:14.392012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:04:14.391979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:28.894075 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:05:28.894038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:05:28.900117 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:05:28.900092    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:05:28.901473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:05:28.901453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:05:28.905546 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:05:28.905524    2579 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:05:28.907849 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:05:28.907817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:06:30.469604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:30.469516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:31.395333 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.395299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"]
Aug 04 18:06:31.395569 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.395557    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" containerName="registry"
Aug 04 18:06:31.395623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.395571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" containerName="registry"
Aug 04 18:06:31.395623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.395615    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cb065b-280b-4d47-b5db-7c0cdfbd43fe" containerName="registry"
Aug 04 18:06:31.397922 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.397903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.400787 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.400766    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:06:31.400917 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.400798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t5sz5\""
Aug 04 18:06:31.401847 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.401828    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:06:31.422404 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.422368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"]
Aug 04 18:06:31.478021 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.477985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.478021 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.478025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.478418 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.478097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp98x\" (UniqueName: \"kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.578424 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.578391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp98x\" (UniqueName: \"kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.578595 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.578454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.578595 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.578476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.578861 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.578843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.578936 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.578884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.587558 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.587541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp98x\" (UniqueName: \"kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.707015 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.706920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:31.831582 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.831544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"]
Aug 04 18:06:31.834038 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:06:31.834000    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f81016_e1d7_46a2_ba8c_86ea97ba1178.slice/crio-49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44 WatchSource:0}: Error finding container 49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44: Status 404 returned error can't find the container with id 49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44
Aug 04 18:06:31.836022 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:31.836007    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:06:32.297085 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:32.297050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l" event={"ID":"25f81016-e1d7-46a2-ba8c-86ea97ba1178","Type":"ContainerStarted","Data":"49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44"}
Aug 04 18:06:33.192807 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:33.192758    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:34.472670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:34.472625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:36.314062 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:36.314025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:36.373703 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:36.373667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:37.316598 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:37.316567    2579 generic.go:356] "Generic (PLEG): container finished" podID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerID="3cbb9278c5614a005c2abbd6d8b688af24cd20ab03f4676d45fa9acb09ca6b32" exitCode=0
Aug 04 18:06:37.316998 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:37.316632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l" event={"ID":"25f81016-e1d7-46a2-ba8c-86ea97ba1178","Type":"ContainerDied","Data":"3cbb9278c5614a005c2abbd6d8b688af24cd20ab03f4676d45fa9acb09ca6b32"}
Aug 04 18:06:41.333088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:41.333049    2579 generic.go:356] "Generic (PLEG): container finished" podID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerID="64942b7e8977a60d0b90a7c146226a2dcd10896028ba6196105a3ff51166220b" exitCode=0
Aug 04 18:06:41.333509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:41.333112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l" event={"ID":"25f81016-e1d7-46a2-ba8c-86ea97ba1178","Type":"ContainerDied","Data":"64942b7e8977a60d0b90a7c146226a2dcd10896028ba6196105a3ff51166220b"}
Aug 04 18:06:47.360693 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:47.360595    2579 generic.go:356] "Generic (PLEG): container finished" podID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerID="bd6942ea09f8f5b5f0d9c25152dc75b59d37e7d31c172021f8bc76502ec44d41" exitCode=0
Aug 04 18:06:47.360693 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:47.360667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l" event={"ID":"25f81016-e1d7-46a2-ba8c-86ea97ba1178","Type":"ContainerDied","Data":"bd6942ea09f8f5b5f0d9c25152dc75b59d37e7d31c172021f8bc76502ec44d41"}
Aug 04 18:06:48.490702 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.490676    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:48.606988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.606946    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp98x\" (UniqueName: \"kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x\") pod \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") "
Aug 04 18:06:48.607182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.607010    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle\") pod \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") "
Aug 04 18:06:48.607182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.607030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util\") pod \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\" (UID: \"25f81016-e1d7-46a2-ba8c-86ea97ba1178\") "
Aug 04 18:06:48.607832 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.607799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle" (OuterVolumeSpecName: "bundle") pod "25f81016-e1d7-46a2-ba8c-86ea97ba1178" (UID: "25f81016-e1d7-46a2-ba8c-86ea97ba1178"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:06:48.609310 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.609292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x" (OuterVolumeSpecName: "kube-api-access-hp98x") pod "25f81016-e1d7-46a2-ba8c-86ea97ba1178" (UID: "25f81016-e1d7-46a2-ba8c-86ea97ba1178"). InnerVolumeSpecName "kube-api-access-hp98x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:06:48.611964 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.611945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util" (OuterVolumeSpecName: "util") pod "25f81016-e1d7-46a2-ba8c-86ea97ba1178" (UID: "25f81016-e1d7-46a2-ba8c-86ea97ba1178"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:06:48.708493 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.708409    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-bundle\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:06:48.708493 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.708438    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25f81016-e1d7-46a2-ba8c-86ea97ba1178-util\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:06:48.708493 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:48.708447    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp98x\" (UniqueName: \"kubernetes.io/projected/25f81016-e1d7-46a2-ba8c-86ea97ba1178-kube-api-access-hp98x\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:06:49.258849 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:49.258814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:49.360008 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:49.359929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:49.369058 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:49.369022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l" event={"ID":"25f81016-e1d7-46a2-ba8c-86ea97ba1178","Type":"ContainerDied","Data":"49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44"}
Aug 04 18:06:49.369058 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:49.369043    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5rrh2l"
Aug 04 18:06:49.369207 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:49.369072    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49139e21e5eaf348df0f3197ef555cef2223654e8c5e060448078a127cee4d44"
Aug 04 18:06:53.374124 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:53.374077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:53.472834 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:53.472796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:58.286321 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.286281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:58.587002 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.586916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:58.601338 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601310    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-g9jk5"]
Aug 04 18:06:58.601561 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601547    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="pull"
Aug 04 18:06:58.601604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="pull"
Aug 04 18:06:58.601604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601578    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="util"
Aug 04 18:06:58.601604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601585    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="util"
Aug 04 18:06:58.601604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="extract"
Aug 04 18:06:58.601604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601596    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="extract"
Aug 04 18:06:58.601773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.601644    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f81016-e1d7-46a2-ba8c-86ea97ba1178" containerName="extract"
Aug 04 18:06:58.643552 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.643527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-g9jk5"]
Aug 04 18:06:58.643686 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.643591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.646316 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.646298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 04 18:06:58.646423 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.646388    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Aug 04 18:06:58.646491 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.646443    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 04 18:06:58.646563 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.646516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Aug 04 18:06:58.647334 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.647309    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-glqs2\""
Aug 04 18:06:58.647395 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.647317    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 04 18:06:58.782137 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.782106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.782295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.782146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5583c200-42f8-4223-b29c-7629042e9c7d-cabundle0\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.782295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.782238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6wj5\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-kube-api-access-m6wj5\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.796856 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.796820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:58.801878 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.801858    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"]
Aug 04 18:06:58.822899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.822872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"]
Aug 04 18:06:58.823054 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.822971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.825750 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.825733    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Aug 04 18:06:58.883215 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37e67610-defd-4029-82e0-b7e949a204c4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.883215 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8q7\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-kube-api-access-lf8q7\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5583c200-42f8-4223-b29c-7629042e9c7d-cabundle0\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wj5\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-kube-api-access-m6wj5\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.883418    2579 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Aug 04 18:06:58.883473 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.883441    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-g9jk5: secret "keda-operator-certs" not found
Aug 04 18:06:58.883788 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.883524    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates podName:5583c200-42f8-4223-b29c-7629042e9c7d nodeName:}" failed. No retries permitted until 2026-08-04 18:06:59.383479537 +0000 UTC m=+390.854766793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates") pod "keda-operator-77f648d694-g9jk5" (UID: "5583c200-42f8-4223-b29c-7629042e9c7d") : secret "keda-operator-certs" not found
Aug 04 18:06:58.883965 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.883947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5583c200-42f8-4223-b29c-7629042e9c7d-cabundle0\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.892748 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.892725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wj5\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-kube-api-access-m6wj5\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:58.984476 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.984438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37e67610-defd-4029-82e0-b7e949a204c4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.984476 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.984474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8q7\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-kube-api-access-lf8q7\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.984669 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.984572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:58.984705 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.984685    2579 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Aug 04 18:06:58.984742 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.984704    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt: secret "keda-metrics-apiserver-certs" not found
Aug 04 18:06:58.984790 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:06:58.984779    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates podName:37e67610-defd-4029-82e0-b7e949a204c4 nodeName:}" failed. No retries permitted until 2026-08-04 18:06:59.484757467 +0000 UTC m=+390.956044739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates") pod "keda-metrics-apiserver-55557df8c-dqpmt" (UID: "37e67610-defd-4029-82e0-b7e949a204c4") : secret "keda-metrics-apiserver-certs" not found
Aug 04 18:06:58.984891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:58.984872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37e67610-defd-4029-82e0-b7e949a204c4-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:59.010367 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.010346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8q7\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-kube-api-access-lf8q7\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:59.038843 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.038816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.133280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.133192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.235075 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.235035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.388057 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.387979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:59.390297 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.390276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5583c200-42f8-4223-b29c-7629042e9c7d-certificates\") pod \"keda-operator-77f648d694-g9jk5\" (UID: \"5583c200-42f8-4223-b29c-7629042e9c7d\") " pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:59.488943 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.488904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:59.491312 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.491291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37e67610-defd-4029-82e0-b7e949a204c4-certificates\") pod \"keda-metrics-apiserver-55557df8c-dqpmt\" (UID: \"37e67610-defd-4029-82e0-b7e949a204c4\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:59.504653 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.504616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.553361 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.553337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:06:59.608600 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.608547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.695570 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.695527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-g9jk5"]
Aug 04 18:06:59.697939 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:06:59.697915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5583c200_42f8_4223_b29c_7629042e9c7d.slice/crio-cc1338bbfc9fa291ca5e0d651ab823d735bdab0a0bc31948a9852c13a1237658 WatchSource:0}: Error finding container cc1338bbfc9fa291ca5e0d651ab823d735bdab0a0bc31948a9852c13a1237658: Status 404 returned error can't find the container with id cc1338bbfc9fa291ca5e0d651ab823d735bdab0a0bc31948a9852c13a1237658
Aug 04 18:06:59.699408 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.699382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.733011 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.732985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:06:59.771021 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.770943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.871420 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.871385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:59.882022 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:06:59.881996    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"]
Aug 04 18:06:59.885324 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:06:59.885287    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e67610_defd_4029_82e0_b7e949a204c4.slice/crio-59acfb0c7f0023f990304be3a4861376dd77c69a2058d3c37f359def418edbb3 WatchSource:0}: Error finding container 59acfb0c7f0023f990304be3a4861376dd77c69a2058d3c37f359def418edbb3: Status 404 returned error can't find the container with id 59acfb0c7f0023f990304be3a4861376dd77c69a2058d3c37f359def418edbb3
Aug 04 18:07:00.408226 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:00.408186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt" event={"ID":"37e67610-defd-4029-82e0-b7e949a204c4","Type":"ContainerStarted","Data":"59acfb0c7f0023f990304be3a4861376dd77c69a2058d3c37f359def418edbb3"}
Aug 04 18:07:00.409298 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:00.409274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-g9jk5" event={"ID":"5583c200-42f8-4223-b29c-7629042e9c7d","Type":"ContainerStarted","Data":"cc1338bbfc9fa291ca5e0d651ab823d735bdab0a0bc31948a9852c13a1237658"}
Aug 04 18:07:04.427431 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.427387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt" event={"ID":"37e67610-defd-4029-82e0-b7e949a204c4","Type":"ContainerStarted","Data":"0df244435e011edb2a79801e8f44bf6d0595a527e07c3a8ab3fa9d208c0b197a"}
Aug 04 18:07:04.427957 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.427588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:07:04.428766 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.428746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-g9jk5" event={"ID":"5583c200-42f8-4223-b29c-7629042e9c7d","Type":"ContainerStarted","Data":"077fc62e0172f123ffbeac2efd28df6f8dc228f8e00e002ad6ded994e782b368"}
Aug 04 18:07:04.428878 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.428867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:07:04.447777 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.447728    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt" podStartSLOduration=2.279149363 podStartE2EDuration="6.447713948s" totalImagesPullingTime="4.168564585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:04.445837928 +0000 UTC m=+395.917125201" watchObservedRunningTime="2026-08-04 18:07:04.447713948 +0000 UTC m=+395.919001200"
Aug 04 18:07:04.463218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:04.463169    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-g9jk5" podStartSLOduration=2.102514109 podStartE2EDuration="6.463152893s" totalImagesPullingTime="4.360638784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:04.462252344 +0000 UTC m=+395.933539644" watchObservedRunningTime="2026-08-04 18:07:04.463152893 +0000 UTC m=+395.934440206"
Aug 04 18:07:15.436535 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:15.436510    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-dqpmt"
Aug 04 18:07:25.434077 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:25.434047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-g9jk5"
Aug 04 18:07:46.129907 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:46.129866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:52.167933 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:52.167847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:52.229852 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:07:52.229821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.240222 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.240184    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:08:19.242299 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.242283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.244381 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.244347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.244747 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.244729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 04 18:08:19.245110 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.245091    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:08:19.245219 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.245204    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:08:19.245266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.245211    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-llk22\""
Aug 04 18:08:19.254034 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.253999    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:08:19.295161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.295129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.295295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.295189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wq29\" (UniqueName: \"kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.315178 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.315147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.371757 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.371707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.395780 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.395751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.395862 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.395800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wq29\" (UniqueName: \"kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.395908 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:08:19.395890    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Aug 04 18:08:19.395971 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:08:19.395960    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert podName:7ba83634-a798-4a3a-a72a-1f8139978da4 nodeName:}" failed. No retries permitted until 2026-08-04 18:08:19.89594481 +0000 UTC m=+471.367232062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert") pod "kserve-controller-manager-5f9d7f69b6-2799p" (UID: "7ba83634-a798-4a3a-a72a-1f8139978da4") : secret "kserve-webhook-server-cert" not found
Aug 04 18:08:19.409912 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.409887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wq29\" (UniqueName: \"kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.468378 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.468341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.877482 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.877447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:19.900012 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.899990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:19.902388 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:19.902365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2799p\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:20.153053 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.152962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:20.276515 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.276478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:08:20.279710 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:08:20.279685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba83634_a798_4a3a_a72a_1f8139978da4.slice/crio-95b6c3bd7f78c028c06a24edb0fd1f5456ce4ab77fad46058984a2464d87d0f4 WatchSource:0}: Error finding container 95b6c3bd7f78c028c06a24edb0fd1f5456ce4ab77fad46058984a2464d87d0f4: Status 404 returned error can't find the container with id 95b6c3bd7f78c028c06a24edb0fd1f5456ce4ab77fad46058984a2464d87d0f4
Aug 04 18:08:20.531545 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.531439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:20.633517 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.633462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:20.686885 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.686854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:20.699461 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:20.699432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" event={"ID":"7ba83634-a798-4a3a-a72a-1f8139978da4","Type":"ContainerStarted","Data":"95b6c3bd7f78c028c06a24edb0fd1f5456ce4ab77fad46058984a2464d87d0f4"}
Aug 04 18:08:23.712678 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:23.712631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" event={"ID":"7ba83634-a798-4a3a-a72a-1f8139978da4","Type":"ContainerStarted","Data":"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"}
Aug 04 18:08:23.713212 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:23.712746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:23.731772 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:23.731723    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" podStartSLOduration=2.252613897 podStartE2EDuration="4.73170682s" totalImagesPullingTime="2.479092923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:23.731573377 +0000 UTC m=+475.202860673" watchObservedRunningTime="2026-08-04 18:08:23.73170682 +0000 UTC m=+475.202994100"
Aug 04 18:08:54.721661 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:54.721629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:08:55.608838 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:08:55.608799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:02.777198 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.777166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:09:02.777599 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.777386    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" podUID="7ba83634-a798-4a3a-a72a-1f8139978da4" containerName="manager" containerID="cri-o://1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722" gracePeriod=10
Aug 04 18:09:02.801964 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.801940    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-x76v9"]
Aug 04 18:09:02.804765 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.804751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.813545 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.813517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c28850-2cc4-4c05-a90c-9690a485ea15-cert\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.813650 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.813603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmz2z\" (UniqueName: \"kubernetes.io/projected/46c28850-2cc4-4c05-a90c-9690a485ea15-kube-api-access-rmz2z\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.814839 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.814818    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-x76v9"]
Aug 04 18:09:02.914393 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.914366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmz2z\" (UniqueName: \"kubernetes.io/projected/46c28850-2cc4-4c05-a90c-9690a485ea15-kube-api-access-rmz2z\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.914538 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.914426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c28850-2cc4-4c05-a90c-9690a485ea15-cert\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.916514 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.916471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c28850-2cc4-4c05-a90c-9690a485ea15-cert\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:02.924458 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:02.924431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmz2z\" (UniqueName: \"kubernetes.io/projected/46c28850-2cc4-4c05-a90c-9690a485ea15-kube-api-access-rmz2z\") pod \"kserve-controller-manager-5f9d7f69b6-x76v9\" (UID: \"46c28850-2cc4-4c05-a90c-9690a485ea15\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:03.013416 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.013394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:09:03.115936 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.115907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wq29\" (UniqueName: \"kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29\") pod \"7ba83634-a798-4a3a-a72a-1f8139978da4\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") "
Aug 04 18:09:03.116073 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.115950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") pod \"7ba83634-a798-4a3a-a72a-1f8139978da4\" (UID: \"7ba83634-a798-4a3a-a72a-1f8139978da4\") "
Aug 04 18:09:03.118114 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.118082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert" (OuterVolumeSpecName: "cert") pod "7ba83634-a798-4a3a-a72a-1f8139978da4" (UID: "7ba83634-a798-4a3a-a72a-1f8139978da4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:09:03.118228 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.118135    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29" (OuterVolumeSpecName: "kube-api-access-5wq29") pod "7ba83634-a798-4a3a-a72a-1f8139978da4" (UID: "7ba83634-a798-4a3a-a72a-1f8139978da4"). InnerVolumeSpecName "kube-api-access-5wq29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:09:03.154293 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.154259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:03.216402 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.216373    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wq29\" (UniqueName: \"kubernetes.io/projected/7ba83634-a798-4a3a-a72a-1f8139978da4-kube-api-access-5wq29\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:09:03.216402 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.216393    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ba83634-a798-4a3a-a72a-1f8139978da4-cert\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:09:03.280449 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.280418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-x76v9"]
Aug 04 18:09:03.283446 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:09:03.283423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c28850_2cc4_4c05_a90c_9690a485ea15.slice/crio-cbe83c741036af754da9726340d2259c266d7e8f6f7858fc52ab4a7dd1eeb201 WatchSource:0}: Error finding container cbe83c741036af754da9726340d2259c266d7e8f6f7858fc52ab4a7dd1eeb201: Status 404 returned error can't find the container with id cbe83c741036af754da9726340d2259c266d7e8f6f7858fc52ab4a7dd1eeb201
Aug 04 18:09:03.852098 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.852060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9" event={"ID":"46c28850-2cc4-4c05-a90c-9690a485ea15","Type":"ContainerStarted","Data":"d564df6950840884218955092a43cb49024b4513272ebc63e9c88ed0f15155a4"}
Aug 04 18:09:03.852451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.852147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:03.852451 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.852173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9" event={"ID":"46c28850-2cc4-4c05-a90c-9690a485ea15","Type":"ContainerStarted","Data":"cbe83c741036af754da9726340d2259c266d7e8f6f7858fc52ab4a7dd1eeb201"}
Aug 04 18:09:03.853277 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.853256    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ba83634-a798-4a3a-a72a-1f8139978da4" containerID="1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722" exitCode=0
Aug 04 18:09:03.853342 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.853285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" event={"ID":"7ba83634-a798-4a3a-a72a-1f8139978da4","Type":"ContainerDied","Data":"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"}
Aug 04 18:09:03.853342 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.853290    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p"
Aug 04 18:09:03.853342 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.853314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2799p" event={"ID":"7ba83634-a798-4a3a-a72a-1f8139978da4","Type":"ContainerDied","Data":"95b6c3bd7f78c028c06a24edb0fd1f5456ce4ab77fad46058984a2464d87d0f4"}
Aug 04 18:09:03.853342 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.853334    2579 scope.go:117] "RemoveContainer" containerID="1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"
Aug 04 18:09:03.861248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.861229    2579 scope.go:117] "RemoveContainer" containerID="1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"
Aug 04 18:09:03.861480 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:09:03.861463    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722\": container with ID starting with 1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722 not found: ID does not exist" containerID="1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"
Aug 04 18:09:03.861554 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.861489    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722"} err="failed to get container status \"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722\": rpc error: code = NotFound desc = could not find container \"1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722\": container with ID starting with 1de8f1d70f644705c3b77bd569a0cd700c37bfb5eeba889993ac905eefdfb722 not found: ID does not exist"
Aug 04 18:09:03.871681 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.871646    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9" podStartSLOduration=1.542883669 podStartE2EDuration="1.871633832s" totalImagesPullingTime="328.750163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:03.871354206 +0000 UTC m=+515.342641480" watchObservedRunningTime="2026-08-04 18:09:03.871633832 +0000 UTC m=+515.342921106"
Aug 04 18:09:03.885486 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.885467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:09:03.890146 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:03.890127    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2799p"]
Aug 04 18:09:05.029450 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:05.029414    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba83634-a798-4a3a-a72a-1f8139978da4" path="/var/lib/kubelet/pods/7ba83634-a798-4a3a-a72a-1f8139978da4/volumes"
Aug 04 18:09:34.862592 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:34.862561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-x76v9"
Aug 04 18:09:37.332569 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.332528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:37.390065 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.390030    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-xr8g9"]
Aug 04 18:09:37.390328 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.390315    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba83634-a798-4a3a-a72a-1f8139978da4" containerName="manager"
Aug 04 18:09:37.390371 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.390330    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba83634-a798-4a3a-a72a-1f8139978da4" containerName="manager"
Aug 04 18:09:37.390408 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.390375    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba83634-a798-4a3a-a72a-1f8139978da4" containerName="manager"
Aug 04 18:09:37.394268 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.394250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.398238 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.398219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Aug 04 18:09:37.399257 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.399236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2r5kg\""
Aug 04 18:09:37.410562 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.410537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xr8g9"]
Aug 04 18:09:37.419295 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.419273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:37.449043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.449013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5kqh\" (UniqueName: \"kubernetes.io/projected/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-kube-api-access-v5kqh\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.449221 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.449090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-tls-certs\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.495371 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.495334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:37.549920 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.549889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-tls-certs\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.550109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.549951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5kqh\" (UniqueName: \"kubernetes.io/projected/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-kube-api-access-v5kqh\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.552385 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.552361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-tls-certs\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.566795 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.566767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5kqh\" (UniqueName: \"kubernetes.io/projected/5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902-kube-api-access-v5kqh\") pod \"model-serving-api-7b954ffc95-xr8g9\" (UID: \"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902\") " pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.569161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.569135    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:37.704336 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.704306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:37.846414 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.846380    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xr8g9"]
Aug 04 18:09:37.849345 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:09:37.849321    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd6d4cc_5d45_4ba6_b2ed_241fbdae8902.slice/crio-a92037c7a9ab015ac8257ce36b764b72243266c15964703e558e6d8471c077c5 WatchSource:0}: Error finding container a92037c7a9ab015ac8257ce36b764b72243266c15964703e558e6d8471c077c5: Status 404 returned error can't find the container with id a92037c7a9ab015ac8257ce36b764b72243266c15964703e558e6d8471c077c5
Aug 04 18:09:37.975248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.975155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xr8g9" event={"ID":"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902","Type":"ContainerStarted","Data":"a92037c7a9ab015ac8257ce36b764b72243266c15964703e558e6d8471c077c5"}
Aug 04 18:09:37.985863 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:37.985834    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:38.318011 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:38.317915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:38.378548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:38.378519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:40.987410 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:40.987372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xr8g9" event={"ID":"5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902","Type":"ContainerStarted","Data":"fe057065d77029eb361664dc064b4b8ed9a2b3625fea572bd25b513b6a03d78e"}
Aug 04 18:09:40.987828 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:40.987451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:09:41.007610 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:41.007562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-xr8g9" podStartSLOduration=1.6984896680000001 podStartE2EDuration="4.007548246s" totalImagesPullingTime="2.309058578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:41.006545239 +0000 UTC m=+552.477832530" watchObservedRunningTime="2026-08-04 18:09:41.007548246 +0000 UTC m=+552.478835519"
Aug 04 18:09:51.996014 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:09:51.995983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-xr8g9"
Aug 04 18:10:28.918241 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:10:28.918214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:10:28.923176 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:10:28.923152    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:10:28.924331 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:10:28.924314    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:10:28.929169 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:10:28.929153    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:11:11.148147 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:11.148107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:11.414708 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:11.414602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.138676 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.138644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.183815 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.183774    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:11:24.185877 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.185861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.188415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.188395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:11:24.188534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.188420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a809-predictor-serving-cert\""
Aug 04 18:11:24.188534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.188396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-64gfv\""
Aug 04 18:11:24.189490 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.189469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4a809-kube-rbac-proxy-sar-config\""
Aug 04 18:11:24.189602 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.189487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:11:24.191159 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.191137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:11:24.346701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.346666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.346701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.346703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzl94\" (UniqueName: \"kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.346916 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.346726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.431277 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.431196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.447733 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.447698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.447908 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.447735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzl94\" (UniqueName: \"kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.447908 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.447764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.448413 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.448389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.450243 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.450224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.457749 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.457719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzl94\" (UniqueName: \"kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94\") pod \"success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") " pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.496998 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.496965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:24.511785 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.511752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.576598 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.576556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.633734 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.633711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:11:24.636566 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:11:24.636539    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75bef20_2245_4b91_95d5_101ebf4adc5f.slice/crio-9c75d9b9b1d60aaae0b3862f46974898e4ff0a50f4ee7f1337ede6f10aaec16c WatchSource:0}: Error finding container 9c75d9b9b1d60aaae0b3862f46974898e4ff0a50f4ee7f1337ede6f10aaec16c: Status 404 returned error can't find the container with id 9c75d9b9b1d60aaae0b3862f46974898e4ff0a50f4ee7f1337ede6f10aaec16c
Aug 04 18:11:24.640899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.640821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.708013 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.707936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.782323 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.782289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.848965 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.848933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:24.949091 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:24.949062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:25.096422 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:25.096347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:25.376004 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:25.375970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerStarted","Data":"9c75d9b9b1d60aaae0b3862f46974898e4ff0a50f4ee7f1337ede6f10aaec16c"}
Aug 04 18:11:35.802863 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:35.802842    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:11:36.430460 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:36.430412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerStarted","Data":"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"}
Aug 04 18:11:38.439819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:38.439785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerStarted","Data":"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"}
Aug 04 18:11:38.440176 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:38.439863    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:38.461272 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:38.461230    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podStartSLOduration=1.268227132 podStartE2EDuration="14.461217728s" totalImagesPullingTime="13.192990596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:11:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:11:38.459695505 +0000 UTC m=+669.930982789" watchObservedRunningTime="2026-08-04 18:11:38.461217728 +0000 UTC m=+669.932504993"
Aug 04 18:11:39.443949 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:39.443915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:39.445266 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:39.445233    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:11:40.314044 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:40.314014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:11:40.447736 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:40.447700    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:11:45.452179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:45.452149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:11:45.452797 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:45.452768    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:11:55.452903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:11:55.452864    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:12:03.367411 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:03.367372    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:04.024453 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:04.024421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:05.452989 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:05.452950    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:12:07.741430 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:07.741395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:14.344209 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:14.344173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:14.444689 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:14.444648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:14.546878 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:14.546850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:15.452701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:15.452661    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:12:25.453693 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:25.453623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:12:44.283857 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.283820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:44.284261 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.283962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:12:44.286159 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.286142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.288710 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.288691    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4a809-kube-rbac-proxy-sar-config\""
Aug 04 18:12:44.288827 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.288694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-4a809-serving-cert\""
Aug 04 18:12:44.296808 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.296786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:12:44.365288 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.365259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.365288 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.365293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.395188 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.395160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:44.465784 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.465752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.465784 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.465789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.466580 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.466563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.468133 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.468115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls\") pod \"switch-graph-4a809-5b56dc99f5-zhq2h\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") " pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.595762 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.595676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:44.722338 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:44.722315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:12:44.725218 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:12:44.725190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8e6585_6401_4757_8a65_7ae36827308f.slice/crio-b48d1d405af20040876a8f6df0b93cb67988570eca01651a3829709bcc422adf WatchSource:0}: Error finding container b48d1d405af20040876a8f6df0b93cb67988570eca01651a3829709bcc422adf: Status 404 returned error can't find the container with id b48d1d405af20040876a8f6df0b93cb67988570eca01651a3829709bcc422adf
Aug 04 18:12:45.682022 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:45.681984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" event={"ID":"0d8e6585-6401-4757-8a65-7ae36827308f","Type":"ContainerStarted","Data":"b48d1d405af20040876a8f6df0b93cb67988570eca01651a3829709bcc422adf"}
Aug 04 18:12:47.690063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:47.690032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" event={"ID":"0d8e6585-6401-4757-8a65-7ae36827308f","Type":"ContainerStarted","Data":"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"}
Aug 04 18:12:47.690442 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:47.690166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:53.700600 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:53.700565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:12:53.720914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:53.720866    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podStartSLOduration=7.13857003 podStartE2EDuration="9.720852268s" totalImagesPullingTime="2.582282238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:47.70997798 +0000 UTC m=+739.181265273" watchObservedRunningTime="2026-08-04 18:12:53.720852268 +0000 UTC m=+745.192139542"
Aug 04 18:12:54.466402 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.466372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:12:54.466605 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.466584    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" containerID="cri-o://1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9" gracePeriod=30
Aug 04 18:12:54.481734 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.481703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.588693 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.588661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.607401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.607369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:12:54.607675 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.607648    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" containerID="cri-o://4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53" gracePeriod=30
Aug 04 18:12:54.607769 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.607686    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kube-rbac-proxy" containerID="cri-o://5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2" gracePeriod=30
Aug 04 18:12:54.654040 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.654014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.654571 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.654551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:12:54.660893 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.660870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.663549 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.663524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-01f07-predictor-serving-cert\""
Aug 04 18:12:54.663667 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.663565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-01f07-kube-rbac-proxy-sar-config\""
Aug 04 18:12:54.669211 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.669191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:12:54.744473 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.744443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.744841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.744482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.744841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.744626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlng7\" (UniqueName: \"kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.755751 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.755725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.845391 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.845355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlng7\" (UniqueName: \"kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.845574 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.845400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.845574 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.845535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.846209 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.846187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.847877 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.847859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.855156 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.855132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.856111 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.856090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlng7\" (UniqueName: \"kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7\") pod \"success-200-isvc-01f07-predictor-678d7fdb78-wxkjk\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") " pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:54.963104 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.963069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:54.972063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:54.972039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:55.103204 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.103174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:12:55.106003 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:12:55.105978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd666f447_d5ed_455b_aa03_63b1359b78bd.slice/crio-1e04b7e1dd9b864568b0b07eb777674055767e73c77e89f858c79d0d2df272d4 WatchSource:0}: Error finding container 1e04b7e1dd9b864568b0b07eb777674055767e73c77e89f858c79d0d2df272d4: Status 404 returned error can't find the container with id 1e04b7e1dd9b864568b0b07eb777674055767e73c77e89f858c79d0d2df272d4
Aug 04 18:12:55.448846 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.448811    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Aug 04 18:12:55.453099 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.453065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Aug 04 18:12:55.723263 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.723189    2579 generic.go:356] "Generic (PLEG): container finished" podID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerID="5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2" exitCode=2
Aug 04 18:12:55.723411 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.723264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerDied","Data":"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"}
Aug 04 18:12:55.725245 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.725221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerStarted","Data":"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"}
Aug 04 18:12:55.725354 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.725276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerStarted","Data":"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"}
Aug 04 18:12:55.725354 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.725306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerStarted","Data":"1e04b7e1dd9b864568b0b07eb777674055767e73c77e89f858c79d0d2df272d4"}
Aug 04 18:12:55.725467 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.725350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:55.746480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:55.746441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podStartSLOduration=1.7464284939999999 podStartE2EDuration="1.746428494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:55.745107916 +0000 UTC m=+747.216395212" watchObservedRunningTime="2026-08-04 18:12:55.746428494 +0000 UTC m=+747.217715825"
Aug 04 18:12:56.729375 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:56.729342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:12:56.730914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:56.730880    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:12:57.336426 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.336401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:12:57.362423 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.362401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzl94\" (UniqueName: \"kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94\") pod \"e75bef20-2245-4b91-95d5-101ebf4adc5f\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") "
Aug 04 18:12:57.362565 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.362479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config\") pod \"e75bef20-2245-4b91-95d5-101ebf4adc5f\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") "
Aug 04 18:12:57.362565 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.362514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls\") pod \"e75bef20-2245-4b91-95d5-101ebf4adc5f\" (UID: \"e75bef20-2245-4b91-95d5-101ebf4adc5f\") "
Aug 04 18:12:57.362822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.362792    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e75bef20-2245-4b91-95d5-101ebf4adc5f" (UID: "e75bef20-2245-4b91-95d5-101ebf4adc5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:12:57.364490 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.364459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e75bef20-2245-4b91-95d5-101ebf4adc5f" (UID: "e75bef20-2245-4b91-95d5-101ebf4adc5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:12:57.364646 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.364531    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94" (OuterVolumeSpecName: "kube-api-access-nzl94") pod "e75bef20-2245-4b91-95d5-101ebf4adc5f" (UID: "e75bef20-2245-4b91-95d5-101ebf4adc5f"). InnerVolumeSpecName "kube-api-access-nzl94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:12:57.463339 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.463280    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzl94\" (UniqueName: \"kubernetes.io/projected/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-api-access-nzl94\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:12:57.463339 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.463311    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e75bef20-2245-4b91-95d5-101ebf4adc5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:12:57.463339 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.463321    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e75bef20-2245-4b91-95d5-101ebf4adc5f-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:12:57.734306 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734241    2579 generic.go:356] "Generic (PLEG): container finished" podID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerID="4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53" exitCode=0
Aug 04 18:12:57.734306 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734297    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"
Aug 04 18:12:57.734444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerDied","Data":"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"}
Aug 04 18:12:57.734444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9" event={"ID":"e75bef20-2245-4b91-95d5-101ebf4adc5f","Type":"ContainerDied","Data":"9c75d9b9b1d60aaae0b3862f46974898e4ff0a50f4ee7f1337ede6f10aaec16c"}
Aug 04 18:12:57.734444 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734408    2579 scope.go:117] "RemoveContainer" containerID="5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"
Aug 04 18:12:57.734956 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.734932    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:12:57.742977 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.742961    2579 scope.go:117] "RemoveContainer" containerID="4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"
Aug 04 18:12:57.749813 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.749797    2579 scope.go:117] "RemoveContainer" containerID="5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"
Aug 04 18:12:57.750052 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:12:57.750036    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2\": container with ID starting with 5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2 not found: ID does not exist" containerID="5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"
Aug 04 18:12:57.750097 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.750057    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2"} err="failed to get container status \"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2\": rpc error: code = NotFound desc = could not find container \"5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2\": container with ID starting with 5fff786f1ff6c552bab072f1927a301982a521dd0d20f245a18042134b61d6a2 not found: ID does not exist"
Aug 04 18:12:57.750097 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.750072    2579 scope.go:117] "RemoveContainer" containerID="4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"
Aug 04 18:12:57.750264 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:12:57.750249    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53\": container with ID starting with 4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53 not found: ID does not exist" containerID="4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"
Aug 04 18:12:57.750304 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.750267    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53"} err="failed to get container status \"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53\": rpc error: code = NotFound desc = could not find container \"4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53\": container with ID starting with 4f312072c168a024f31e41fee31f49b66e15954f09b60588d3324047bcd99c53 not found: ID does not exist"
Aug 04 18:12:57.758115 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.758094    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:12:57.760916 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:57.760898    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4a809-predictor-5dcd77f4d6-v75w9"]
Aug 04 18:12:58.698051 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:58.698012    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:12:59.033836 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:12:59.033759    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" path="/var/lib/kubelet/pods/e75bef20-2245-4b91-95d5-101ebf4adc5f/volumes"
Aug 04 18:13:02.740132 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:02.740101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:13:02.740670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:02.740646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:13:03.697956 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:03.697914    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:08.699042 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:08.699001    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:08.699490 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:08.699119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:13:12.740630 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:12.740590    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:13:13.698893 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:13.698852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:14.048585 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:14.048489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:18.698480 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:18.698441    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:22.740846 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:22.740806    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:13:23.698571 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:23.698526    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:24.612391 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.612370    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:13:24.672971 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.672945    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls\") pod \"0d8e6585-6401-4757-8a65-7ae36827308f\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") "
Aug 04 18:13:24.673080 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.672987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle\") pod \"0d8e6585-6401-4757-8a65-7ae36827308f\" (UID: \"0d8e6585-6401-4757-8a65-7ae36827308f\") "
Aug 04 18:13:24.673358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.673338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d8e6585-6401-4757-8a65-7ae36827308f" (UID: "0d8e6585-6401-4757-8a65-7ae36827308f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:13:24.674874 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.674856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d8e6585-6401-4757-8a65-7ae36827308f" (UID: "0d8e6585-6401-4757-8a65-7ae36827308f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:13:24.774280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.774219    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8e6585-6401-4757-8a65-7ae36827308f-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:13:24.774280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.774243    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8e6585-6401-4757-8a65-7ae36827308f-openshift-service-ca-bundle\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:13:24.843756 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.843723    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d8e6585-6401-4757-8a65-7ae36827308f" containerID="1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9" exitCode=0
Aug 04 18:13:24.843887 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.843786    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"
Aug 04 18:13:24.843887 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.843805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" event={"ID":"0d8e6585-6401-4757-8a65-7ae36827308f","Type":"ContainerDied","Data":"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"}
Aug 04 18:13:24.843887 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.843874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h" event={"ID":"0d8e6585-6401-4757-8a65-7ae36827308f","Type":"ContainerDied","Data":"b48d1d405af20040876a8f6df0b93cb67988570eca01651a3829709bcc422adf"}
Aug 04 18:13:24.844045 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.843897    2579 scope.go:117] "RemoveContainer" containerID="1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"
Aug 04 18:13:24.851654 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.851629    2579 scope.go:117] "RemoveContainer" containerID="1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"
Aug 04 18:13:24.851895 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:13:24.851877    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9\": container with ID starting with 1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9 not found: ID does not exist" containerID="1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"
Aug 04 18:13:24.851952 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.851901    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9"} err="failed to get container status \"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9\": rpc error: code = NotFound desc = could not find container \"1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9\": container with ID starting with 1b58a46272e4a096891cebaa84cfeabb677dd8b4783a464c3dbd448549cb3df9 not found: ID does not exist"
Aug 04 18:13:24.867914 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.867894    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:13:24.875232 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:24.875215    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-4a809-5b56dc99f5-zhq2h"]
Aug 04 18:13:25.030179 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:25.030114    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" path="/var/lib/kubelet/pods/0d8e6585-6401-4757-8a65-7ae36827308f/volumes"
Aug 04 18:13:32.741178 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:32.741124    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Aug 04 18:13:42.741662 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:42.741633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:13:44.265921 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.265867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:44.269491 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269465    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:13:44.269756 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269743    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809"
Aug 04 18:13:44.269798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269758    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809"
Aug 04 18:13:44.269798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269777    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container"
Aug 04 18:13:44.269798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container"
Aug 04 18:13:44.269798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269791    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kube-rbac-proxy"
Aug 04 18:13:44.269798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kube-rbac-proxy"
Aug 04 18:13:44.269945 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269845    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8e6585-6401-4757-8a65-7ae36827308f" containerName="switch-graph-4a809"
Aug 04 18:13:44.269945 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269854    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kube-rbac-proxy"
Aug 04 18:13:44.269945 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.269862    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75bef20-2245-4b91-95d5-101ebf4adc5f" containerName="kserve-container"
Aug 04 18:13:44.272814 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.272798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.275597 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.275573    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Aug 04 18:13:44.275699 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.275615    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Aug 04 18:13:44.283182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.283161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:13:44.419352 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.419312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.419564 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.419399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.520549 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.520435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.520549 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.520488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.520735 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:13:44.520625    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Aug 04 18:13:44.520735 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:13:44.520713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls podName:a256c7df-7cb4-4e49-9461-3cee9bcf48f4 nodeName:}" failed. No retries permitted until 2026-08-04 18:13:45.020691579 +0000 UTC m=+796.491978843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls") pod "model-chainer-644bd56996-mww9p" (UID: "a256c7df-7cb4-4e49-9461-3cee9bcf48f4") : secret "model-chainer-serving-cert" not found
Aug 04 18:13:44.521223 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.521201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:44.947477 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:44.947446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:45.025546 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.025516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:45.028044 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.028017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") pod \"model-chainer-644bd56996-mww9p\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") " pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:45.183664 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.183631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:45.307172 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.307017    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:13:45.309906 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:13:45.309875    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda256c7df_7cb4_4e49_9461_3cee9bcf48f4.slice/crio-f390d42e20a234834584b116aa1920756598427768986b16444e05fbf5dbfe56 WatchSource:0}: Error finding container f390d42e20a234834584b116aa1920756598427768986b16444e05fbf5dbfe56: Status 404 returned error can't find the container with id f390d42e20a234834584b116aa1920756598427768986b16444e05fbf5dbfe56
Aug 04 18:13:45.932222 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.932178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" event={"ID":"a256c7df-7cb4-4e49-9461-3cee9bcf48f4","Type":"ContainerStarted","Data":"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"}
Aug 04 18:13:45.932392 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.932244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" event={"ID":"a256c7df-7cb4-4e49-9461-3cee9bcf48f4","Type":"ContainerStarted","Data":"f390d42e20a234834584b116aa1920756598427768986b16444e05fbf5dbfe56"}
Aug 04 18:13:45.932392 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.932312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:45.954304 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:45.954252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podStartSLOduration=1.9542378660000002 podStartE2EDuration="1.954237866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:45.953942513 +0000 UTC m=+797.425229786" watchObservedRunningTime="2026-08-04 18:13:45.954237866 +0000 UTC m=+797.425525139"
Aug 04 18:13:51.942413 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:51.942340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:13:54.373252 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.373222    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:13:54.373725 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.373389    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" containerID="cri-o://1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4" gracePeriod=30
Aug 04 18:13:54.390663 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.390628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.488244 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.488202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.552119 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.552092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.560407 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.560377    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:13:54.563371 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.563355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.565955 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.565919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-315c2-predictor-serving-cert\""
Aug 04 18:13:54.566178 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.566163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-315c2-kube-rbac-proxy-sar-config\""
Aug 04 18:13:54.575229 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.575200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:13:54.656151 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.656078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.696303 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.696269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.696303 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.696303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4l4\" (UniqueName: \"kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.696564 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.696406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.729246 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.729203    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:13:54.733049 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.732761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:54.734413 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.734389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.735807 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.735790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-01f07-serving-cert\""
Aug 04 18:13:54.735807 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.735796    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-01f07-kube-rbac-proxy-sar-config\""
Aug 04 18:13:54.741015 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.740993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:13:54.792915 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.792869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.797152 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.797123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.797272 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.797162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4l4\" (UniqueName: \"kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.797272 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.797236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.797815 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.797793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.799891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.799861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.806391 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.806368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4l4\" (UniqueName: \"kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4\") pod \"success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") " pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.873608 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.873575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:54.893969 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.893941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:54.898014 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.897991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:54.898105 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.898040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:54.999338 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.999307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:54.999506 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.999352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:54.999939 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:54.999919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:55.003176 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.003151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls\") pod \"switch-graph-01f07-6d5fcffdb-85b7c\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") " pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:55.003923 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.003905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:13:55.006867 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:13:55.006844    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a17e45_953b_43f4_be9f_015b64efddff.slice/crio-42f2ff334c05bac4e318e9e4d67299b073af2f4e3545eb14004b29896ed638f8 WatchSource:0}: Error finding container 42f2ff334c05bac4e318e9e4d67299b073af2f4e3545eb14004b29896ed638f8: Status 404 returned error can't find the container with id 42f2ff334c05bac4e318e9e4d67299b073af2f4e3545eb14004b29896ed638f8
Aug 04 18:13:55.043877 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.043853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:55.180813 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.180694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:13:55.183268 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:13:55.183242    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956e704a_aa32_46ef_bd60_39e4bcddd4f1.slice/crio-98b0710b8d153980f61bf8bd39aa9a1adba8617f2ef69ac7c7bcb6330831069d WatchSource:0}: Error finding container 98b0710b8d153980f61bf8bd39aa9a1adba8617f2ef69ac7c7bcb6330831069d: Status 404 returned error can't find the container with id 98b0710b8d153980f61bf8bd39aa9a1adba8617f2ef69ac7c7bcb6330831069d
Aug 04 18:13:55.971548 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.971486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerStarted","Data":"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"}
Aug 04 18:13:55.971988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.971579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerStarted","Data":"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"}
Aug 04 18:13:55.971988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.971606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerStarted","Data":"42f2ff334c05bac4e318e9e4d67299b073af2f4e3545eb14004b29896ed638f8"}
Aug 04 18:13:55.971988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.971640    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:55.971988 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.971663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:13:55.972963 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.972936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" event={"ID":"956e704a-aa32-46ef-bd60-39e4bcddd4f1","Type":"ContainerStarted","Data":"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"}
Aug 04 18:13:55.973060 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.972986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" event={"ID":"956e704a-aa32-46ef-bd60-39e4bcddd4f1","Type":"ContainerStarted","Data":"98b0710b8d153980f61bf8bd39aa9a1adba8617f2ef69ac7c7bcb6330831069d"}
Aug 04 18:13:55.973060 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.973032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:13:55.973146 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.973125    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:13:55.991011 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:55.990957    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podStartSLOduration=1.990941802 podStartE2EDuration="1.990941802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:55.989384443 +0000 UTC m=+807.460671737" watchObservedRunningTime="2026-08-04 18:13:55.990941802 +0000 UTC m=+807.462229076"
Aug 04 18:13:56.005155 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:56.005108    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podStartSLOduration=2.005093612 podStartE2EDuration="2.005093612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:56.004436334 +0000 UTC m=+807.475723605" watchObservedRunningTime="2026-08-04 18:13:56.005093612 +0000 UTC m=+807.476380886"
Aug 04 18:13:56.940558 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:56.940519    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:13:56.977858 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:13:56.977822    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:14:01.940649 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:01.940609    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:01.983146 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:01.983121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:14:01.983629 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:01.983603    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:14:01.984717 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:01.984697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:14:06.940621 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:06.940576    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:06.940997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:06.940685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:14:11.940401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:11.940356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:11.984305 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:11.984272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:14:16.940246 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:16.940203    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:21.941842 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:21.941797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:14:21.984349 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:21.984318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:14:24.515299 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.515271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:14:24.608037 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.607999    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") pod \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") "
Aug 04 18:14:24.608037 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.608044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle\") pod \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\" (UID: \"a256c7df-7cb4-4e49-9461-3cee9bcf48f4\") "
Aug 04 18:14:24.608421 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.608388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a256c7df-7cb4-4e49-9461-3cee9bcf48f4" (UID: "a256c7df-7cb4-4e49-9461-3cee9bcf48f4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:24.610037 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.610005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a256c7df-7cb4-4e49-9461-3cee9bcf48f4" (UID: "a256c7df-7cb4-4e49-9461-3cee9bcf48f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:24.709300 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.709219    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:14:24.709300 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:24.709246    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a256c7df-7cb4-4e49-9461-3cee9bcf48f4-openshift-service-ca-bundle\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:14:25.081741 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.081710    2579 generic.go:356] "Generic (PLEG): container finished" podID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerID="1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4" exitCode=0
Aug 04 18:14:25.081903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.081755    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"
Aug 04 18:14:25.081903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.081777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" event={"ID":"a256c7df-7cb4-4e49-9461-3cee9bcf48f4","Type":"ContainerDied","Data":"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"}
Aug 04 18:14:25.081903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.081818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-644bd56996-mww9p" event={"ID":"a256c7df-7cb4-4e49-9461-3cee9bcf48f4","Type":"ContainerDied","Data":"f390d42e20a234834584b116aa1920756598427768986b16444e05fbf5dbfe56"}
Aug 04 18:14:25.081903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.081840    2579 scope.go:117] "RemoveContainer" containerID="1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"
Aug 04 18:14:25.089453 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.089435    2579 scope.go:117] "RemoveContainer" containerID="1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"
Aug 04 18:14:25.089685 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:14:25.089667    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4\": container with ID starting with 1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4 not found: ID does not exist" containerID="1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"
Aug 04 18:14:25.089732 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.089695    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4"} err="failed to get container status \"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4\": rpc error: code = NotFound desc = could not find container \"1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4\": container with ID starting with 1e00456c13c680cbe0ca6b88467fea5c7e468ac0a53c9324365d4237054d5ee4 not found: ID does not exist"
Aug 04 18:14:25.100576 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.100554    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:14:25.106560 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:25.106534    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-644bd56996-mww9p"]
Aug 04 18:14:27.030182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:27.030150    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" path="/var/lib/kubelet/pods/a256c7df-7cb4-4e49-9461-3cee9bcf48f4/volumes"
Aug 04 18:14:31.984471 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:31.984432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:14:41.984679 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:41.984644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:14:54.587984 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:54.587945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:55.110961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:14:55.110933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:28.942175 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:15:28.942098    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:15:28.944992 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:15:28.944970    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:15:28.948422 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:15:28.948401    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:15:28.951218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:15:28.951200    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:16:30.494635 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:16:30.494589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:34.864904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:17:34.864863    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:14.400517 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:18:14.400453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:14.566890 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:18:14.566854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:28.965314 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:20:28.965282    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:20:28.966876 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:20:28.966855    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:20:28.971028 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:20:28.970996    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:20:28.972759 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:20:28.972741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:22:09.459148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.459112    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:22:09.461636 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.459370    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" containerID="cri-o://62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea" gracePeriod=30
Aug 04 18:22:09.507375 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.507347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:09.610520 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.610467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:22:09.610773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.610741    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container" containerID="cri-o://63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157" gracePeriod=30
Aug 04 18:22:09.610894 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.610789    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kube-rbac-proxy" containerID="cri-o://833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df" gracePeriod=30
Aug 04 18:22:09.639319 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.639297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:09.684770 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.684746    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:22:09.685036 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.685022    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer"
Aug 04 18:22:09.685036 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.685036    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer"
Aug 04 18:22:09.685168 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.685098    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a256c7df-7cb4-4e49-9461-3cee9bcf48f4" containerName="model-chainer"
Aug 04 18:22:09.688382 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.688363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.690997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.690958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1525-kube-rbac-proxy-sar-config\""
Aug 04 18:22:09.691107 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.691041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1525-predictor-serving-cert\""
Aug 04 18:22:09.704060 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.704038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:22:09.714673 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.714611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:09.792872 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.792824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:09.798196 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.798175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.798285 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.798210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.798285 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.798236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl8s\" (UniqueName: \"kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.816008 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.815841    2579 generic.go:356] "Generic (PLEG): container finished" podID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerID="833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df" exitCode=2
Aug 04 18:22:09.816008 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.815916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerDied","Data":"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"}
Aug 04 18:22:09.898797 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.898767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.898962 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.898803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.898962 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.898827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl8s\" (UniqueName: \"kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.899439 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.899415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.901481 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.901457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:09.908358 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:09.908339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl8s\" (UniqueName: \"kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s\") pod \"success-200-isvc-a1525-predictor-596bfc45bb-kpntg\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") " pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:10.000300 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.000211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:10.051991 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.051953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:10.130665 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.130640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:22:10.133633 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:22:10.133608    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6451404_5d8d_491a_bfaf_6914ec87bb09.slice/crio-ec715758034f4c9c0e44f029656ca81e16f78c53e61360c870c8a4a7200ba008 WatchSource:0}: Error finding container ec715758034f4c9c0e44f029656ca81e16f78c53e61360c870c8a4a7200ba008: Status 404 returned error can't find the container with id ec715758034f4c9c0e44f029656ca81e16f78c53e61360c870c8a4a7200ba008
Aug 04 18:22:10.821416 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.821384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerStarted","Data":"6b8f1e82c85fab45a988338f00eaf39ba1855c13181e5e294eb8e4cde4f935e9"}
Aug 04 18:22:10.821810 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.821437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerStarted","Data":"ff1396b698144a5a28121380c7b70ccb88953c1687d45ce5603017c0f058b8be"}
Aug 04 18:22:10.821810 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.821458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerStarted","Data":"ec715758034f4c9c0e44f029656ca81e16f78c53e61360c870c8a4a7200ba008"}
Aug 04 18:22:10.821810 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.821538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:10.841436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:10.841392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podStartSLOduration=1.8413745449999999 podStartE2EDuration="1.841374545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:10.841358417 +0000 UTC m=+1302.312645690" watchObservedRunningTime="2026-08-04 18:22:10.841374545 +0000 UTC m=+1302.312661820"
Aug 04 18:22:11.824830 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:11.824792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:11.826068 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:11.826040    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:11.981923 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:11.981885    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:12.657810 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.657785    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:22:12.825205 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.825115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlng7\" (UniqueName: \"kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7\") pod \"d666f447-d5ed-455b-aa03-63b1359b78bd\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") "
Aug 04 18:22:12.825205 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.825166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config\") pod \"d666f447-d5ed-455b-aa03-63b1359b78bd\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") "
Aug 04 18:22:12.825205 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.825196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls\") pod \"d666f447-d5ed-455b-aa03-63b1359b78bd\" (UID: \"d666f447-d5ed-455b-aa03-63b1359b78bd\") "
Aug 04 18:22:12.825728 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.825573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d666f447-d5ed-455b-aa03-63b1359b78bd" (UID: "d666f447-d5ed-455b-aa03-63b1359b78bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:12.827381 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.827343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d666f447-d5ed-455b-aa03-63b1359b78bd" (UID: "d666f447-d5ed-455b-aa03-63b1359b78bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:12.827484 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.827394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7" (OuterVolumeSpecName: "kube-api-access-tlng7") pod "d666f447-d5ed-455b-aa03-63b1359b78bd" (UID: "d666f447-d5ed-455b-aa03-63b1359b78bd"). InnerVolumeSpecName "kube-api-access-tlng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:12.830942 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.830910    2579 generic.go:356] "Generic (PLEG): container finished" podID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerID="63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157" exitCode=0
Aug 04 18:22:12.831057 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.830977    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"
Aug 04 18:22:12.831057 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.830984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerDied","Data":"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"}
Aug 04 18:22:12.831184 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.831098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk" event={"ID":"d666f447-d5ed-455b-aa03-63b1359b78bd","Type":"ContainerDied","Data":"1e04b7e1dd9b864568b0b07eb777674055767e73c77e89f858c79d0d2df272d4"}
Aug 04 18:22:12.831184 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.831141    2579 scope.go:117] "RemoveContainer" containerID="833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"
Aug 04 18:22:12.831690 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.831662    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:12.842891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.842874    2579 scope.go:117] "RemoveContainer" containerID="63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"
Aug 04 18:22:12.849941 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.849923    2579 scope.go:117] "RemoveContainer" containerID="833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"
Aug 04 18:22:12.850184 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:22:12.850166    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df\": container with ID starting with 833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df not found: ID does not exist" containerID="833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"
Aug 04 18:22:12.850239 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.850192    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df"} err="failed to get container status \"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df\": rpc error: code = NotFound desc = could not find container \"833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df\": container with ID starting with 833173c76dff021cf783fc395610b7ec7949fade53914b2e8a103e7029f1b2df not found: ID does not exist"
Aug 04 18:22:12.850239 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.850208    2579 scope.go:117] "RemoveContainer" containerID="63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"
Aug 04 18:22:12.850403 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:22:12.850390    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157\": container with ID starting with 63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157 not found: ID does not exist" containerID="63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"
Aug 04 18:22:12.850438 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.850407    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157"} err="failed to get container status \"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157\": rpc error: code = NotFound desc = could not find container \"63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157\": container with ID starting with 63e495a2f3b9f82a02c42cda8892153acd557218bcddca016fd4c862b50fd157 not found: ID does not exist"
Aug 04 18:22:12.857352 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.857332    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:22:12.862305 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.862284    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-01f07-predictor-678d7fdb78-wxkjk"]
Aug 04 18:22:12.926250 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.926227    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlng7\" (UniqueName: \"kubernetes.io/projected/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-api-access-tlng7\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:22:12.926250 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.926246    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d666f447-d5ed-455b-aa03-63b1359b78bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:22:12.926401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:12.926256    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d666f447-d5ed-455b-aa03-63b1359b78bd-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:22:13.029597 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:13.029569    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" path="/var/lib/kubelet/pods/d666f447-d5ed-455b-aa03-63b1359b78bd/volumes"
Aug 04 18:22:16.981746 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:16.981712    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:17.835888 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:17.835855    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:22:17.836445 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:17.836420    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:21.981756 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:21.981721    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:21.982165 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:21.981823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:22:26.987005 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:26.986949    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:27.048947 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:27.048920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:27.837166 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:27.837126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:31.981745 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:31.981708    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:36.982921 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:36.982878    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 04 18:22:37.836651 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:37.836616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:39.593627 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.593603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:22:39.635674 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.635649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle\") pod \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") "
Aug 04 18:22:39.635801 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.635701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls\") pod \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\" (UID: \"956e704a-aa32-46ef-bd60-39e4bcddd4f1\") "
Aug 04 18:22:39.636014 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.635994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "956e704a-aa32-46ef-bd60-39e4bcddd4f1" (UID: "956e704a-aa32-46ef-bd60-39e4bcddd4f1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:22:39.637733 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.637713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "956e704a-aa32-46ef-bd60-39e4bcddd4f1" (UID: "956e704a-aa32-46ef-bd60-39e4bcddd4f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:39.736809 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.736737    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956e704a-aa32-46ef-bd60-39e4bcddd4f1-openshift-service-ca-bundle\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:22:39.736809 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.736771    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956e704a-aa32-46ef-bd60-39e4bcddd4f1-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:22:39.937329 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.937291    2579 generic.go:356] "Generic (PLEG): container finished" podID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerID="62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea" exitCode=0
Aug 04 18:22:39.937329 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.937331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"
Aug 04 18:22:39.937567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.937342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" event={"ID":"956e704a-aa32-46ef-bd60-39e4bcddd4f1","Type":"ContainerDied","Data":"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"}
Aug 04 18:22:39.937567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.937399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c" event={"ID":"956e704a-aa32-46ef-bd60-39e4bcddd4f1","Type":"ContainerDied","Data":"98b0710b8d153980f61bf8bd39aa9a1adba8617f2ef69ac7c7bcb6330831069d"}
Aug 04 18:22:39.937567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.937430    2579 scope.go:117] "RemoveContainer" containerID="62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"
Aug 04 18:22:39.945895 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.945875    2579 scope.go:117] "RemoveContainer" containerID="62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"
Aug 04 18:22:39.946144 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:22:39.946126    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea\": container with ID starting with 62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea not found: ID does not exist" containerID="62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"
Aug 04 18:22:39.946191 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.946153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea"} err="failed to get container status \"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea\": rpc error: code = NotFound desc = could not find container \"62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea\": container with ID starting with 62e6c9869bec4e6e4f65f8cfa55a496ca7fc356a0708c857b5d96f7edc3e64ea not found: ID does not exist"
Aug 04 18:22:39.960132 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.960109    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:22:39.964997 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:39.964977    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-01f07-6d5fcffdb-85b7c"]
Aug 04 18:22:41.030148 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:41.030105    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" path="/var/lib/kubelet/pods/956e704a-aa32-46ef-bd60-39e4bcddd4f1/volumes"
Aug 04 18:22:47.837350 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:47.837310    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Aug 04 18:22:57.837713 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:22:57.837632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:23:09.359271 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.359232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:09.468932 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.468897    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:23:09.469178 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.469130    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" containerID="cri-o://0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca" gracePeriod=30
Aug 04 18:23:09.469240 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.469184    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kube-rbac-proxy" containerID="cri-o://c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c" gracePeriod=30
Aug 04 18:23:09.540171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540136    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:23:09.540464 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540451    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540487    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kube-rbac-proxy"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kube-rbac-proxy"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="956e704a-aa32-46ef-bd60-39e4bcddd4f1" containerName="switch-graph-01f07"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540579    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kserve-container"
Aug 04 18:23:09.540819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.540586    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d666f447-d5ed-455b-aa03-63b1359b78bd" containerName="kube-rbac-proxy"
Aug 04 18:23:09.543773 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.543752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.546168 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.546143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-94d00-kube-rbac-proxy-sar-config\""
Aug 04 18:23:09.546283 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.546143    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-94d00-predictor-serving-cert\""
Aug 04 18:23:09.554248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.554221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:23:09.681652 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.681617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq56k\" (UniqueName: \"kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.681652 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.681658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.681904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.681749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.782826 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.782788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.782995 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.782839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq56k\" (UniqueName: \"kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.782995 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:23:09.782933    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-94d00-predictor-serving-cert: secret "success-200-isvc-94d00-predictor-serving-cert" not found
Aug 04 18:23:09.782995 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.782956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.783115 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:23:09.783018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls podName:41d6b9b8-e816-4e5b-a834-9149bbfcc784 nodeName:}" failed. No retries permitted until 2026-08-04 18:23:10.282990829 +0000 UTC m=+1361.754278081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls") pod "success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" (UID: "41d6b9b8-e816-4e5b-a834-9149bbfcc784") : secret "success-200-isvc-94d00-predictor-serving-cert" not found
Aug 04 18:23:09.783622 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.783606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:09.792333 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:09.792299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq56k\" (UniqueName: \"kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:10.051122 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.051031    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1a17e45-953b-43f4-be9f-015b64efddff" containerID="c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c" exitCode=2
Aug 04 18:23:10.051263 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.051111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerDied","Data":"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"}
Aug 04 18:23:10.287618 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.287575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:10.289928 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.289909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") pod \"success-200-isvc-94d00-predictor-85bcffb47b-j4wcf\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") " pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:10.424913 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.424864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:10.454761 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.454726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:10.589695 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:10.589665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:23:10.592042 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:23:10.592014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d6b9b8_e816_4e5b_a834_9149bbfcc784.slice/crio-374627b1bb9fca6d6671da8ed9fe1dea1c8a56215dc61f3739aac8335a223b77 WatchSource:0}: Error finding container 374627b1bb9fca6d6671da8ed9fe1dea1c8a56215dc61f3739aac8335a223b77: Status 404 returned error can't find the container with id 374627b1bb9fca6d6671da8ed9fe1dea1c8a56215dc61f3739aac8335a223b77
Aug 04 18:23:11.057207 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.057120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerStarted","Data":"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"}
Aug 04 18:23:11.057207 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.057183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerStarted","Data":"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"}
Aug 04 18:23:11.057390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.057219    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:11.057390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.057240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerStarted","Data":"374627b1bb9fca6d6671da8ed9fe1dea1c8a56215dc61f3739aac8335a223b77"}
Aug 04 18:23:11.076048 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.075996    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podStartSLOduration=2.075976131 podStartE2EDuration="2.075976131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:11.07525685 +0000 UTC m=+1362.546544123" watchObservedRunningTime="2026-08-04 18:23:11.075976131 +0000 UTC m=+1362.547263407"
Aug 04 18:23:11.978535 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.978478    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Aug 04 18:23:11.983627 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:11.983598    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Aug 04 18:23:12.063556 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.063527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:12.064822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.064791    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:12.529588 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.529557    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:23:12.709754 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.709708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config\") pod \"e1a17e45-953b-43f4-be9f-015b64efddff\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") "
Aug 04 18:23:12.709754 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.709762    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls\") pod \"e1a17e45-953b-43f4-be9f-015b64efddff\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") "
Aug 04 18:23:12.710024 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.709811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs4l4\" (UniqueName: \"kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4\") pod \"e1a17e45-953b-43f4-be9f-015b64efddff\" (UID: \"e1a17e45-953b-43f4-be9f-015b64efddff\") "
Aug 04 18:23:12.710172 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.710145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1a17e45-953b-43f4-be9f-015b64efddff" (UID: "e1a17e45-953b-43f4-be9f-015b64efddff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:12.711904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.711873    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1a17e45-953b-43f4-be9f-015b64efddff" (UID: "e1a17e45-953b-43f4-be9f-015b64efddff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:12.711904 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.711879    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4" (OuterVolumeSpecName: "kube-api-access-fs4l4") pod "e1a17e45-953b-43f4-be9f-015b64efddff" (UID: "e1a17e45-953b-43f4-be9f-015b64efddff"). InnerVolumeSpecName "kube-api-access-fs4l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:12.810679 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.810628    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4l4\" (UniqueName: \"kubernetes.io/projected/e1a17e45-953b-43f4-be9f-015b64efddff-kube-api-access-fs4l4\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:12.810679 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.810674    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a17e45-953b-43f4-be9f-015b64efddff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:12.810679 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:12.810686    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a17e45-953b-43f4-be9f-015b64efddff-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:13.068543 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.068488    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1a17e45-953b-43f4-be9f-015b64efddff" containerID="0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca" exitCode=0
Aug 04 18:23:13.068985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.068552    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"
Aug 04 18:23:13.068985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.068567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerDied","Data":"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"}
Aug 04 18:23:13.068985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.068626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq" event={"ID":"e1a17e45-953b-43f4-be9f-015b64efddff","Type":"ContainerDied","Data":"42f2ff334c05bac4e318e9e4d67299b073af2f4e3545eb14004b29896ed638f8"}
Aug 04 18:23:13.068985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.068651    2579 scope.go:117] "RemoveContainer" containerID="c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"
Aug 04 18:23:13.069165 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.069141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:13.077134 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.077115    2579 scope.go:117] "RemoveContainer" containerID="0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"
Aug 04 18:23:13.084507 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.084476    2579 scope.go:117] "RemoveContainer" containerID="c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"
Aug 04 18:23:13.084780 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:23:13.084758    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c\": container with ID starting with c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c not found: ID does not exist" containerID="c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"
Aug 04 18:23:13.084888 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.084789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c"} err="failed to get container status \"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c\": rpc error: code = NotFound desc = could not find container \"c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c\": container with ID starting with c4e9366748a9743a1c3b0063e4ec7404d073956a2361f3b75ba0032c01d3311c not found: ID does not exist"
Aug 04 18:23:13.084888 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.084810    2579 scope.go:117] "RemoveContainer" containerID="0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"
Aug 04 18:23:13.085144 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:23:13.085123    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca\": container with ID starting with 0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca not found: ID does not exist" containerID="0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"
Aug 04 18:23:13.085244 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.085153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca"} err="failed to get container status \"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca\": rpc error: code = NotFound desc = could not find container \"0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca\": container with ID starting with 0ae762b7b71b8b839d08e1079597971f1462885335408e6fa973542b3ce7f0ca not found: ID does not exist"
Aug 04 18:23:13.086701 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.086682    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:23:13.091188 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:13.091167    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-315c2-predictor-69cd9f9dfd-h8vcq"]
Aug 04 18:23:15.029289 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:15.029258    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" path="/var/lib/kubelet/pods/e1a17e45-953b-43f4-be9f-015b64efddff/volumes"
Aug 04 18:23:18.073607 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:18.073581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:23:18.074073 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:18.074015    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:19.803802 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.803722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:19.922796 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.922758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:23:19.923115 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.923087    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container" containerID="cri-o://ff1396b698144a5a28121380c7b70ccb88953c1687d45ce5603017c0f058b8be" gracePeriod=30
Aug 04 18:23:19.923220 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.923151    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kube-rbac-proxy" containerID="cri-o://6b8f1e82c85fab45a988338f00eaf39ba1855c13181e5e294eb8e4cde4f935e9" gracePeriod=30
Aug 04 18:23:19.951725 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.951692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:19.993405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993376    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:23:19.993742 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993727    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container"
Aug 04 18:23:19.993791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container"
Aug 04 18:23:19.993791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993758    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kube-rbac-proxy"
Aug 04 18:23:19.993791 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993764    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kube-rbac-proxy"
Aug 04 18:23:19.993882 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993821    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kserve-container"
Aug 04 18:23:19.993882 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.993830    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a17e45-953b-43f4-be9f-015b64efddff" containerName="kube-rbac-proxy"
Aug 04 18:23:19.996841 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.996823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:19.999217 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.999183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a162f-predictor-serving-cert\""
Aug 04 18:23:19.999306 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:19.999281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a162f-kube-rbac-proxy-sar-config\""
Aug 04 18:23:20.015390 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.015367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:23:20.043402 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.043377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:20.097458 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.097384    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerID="6b8f1e82c85fab45a988338f00eaf39ba1855c13181e5e294eb8e4cde4f935e9" exitCode=2
Aug 04 18:23:20.097593 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.097453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerDied","Data":"6b8f1e82c85fab45a988338f00eaf39ba1855c13181e5e294eb8e4cde4f935e9"}
Aug 04 18:23:20.115361 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.115336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:20.173383 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.173347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.173383 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.173382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.173639 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.173414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85mj\" (UniqueName: \"kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.236999 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.236969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:20.274220 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.274187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.274220 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.274223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.274393 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.274270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85mj\" (UniqueName: \"kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.274943 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.274922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.276883 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.276864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.283327 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.283301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85mj\" (UniqueName: \"kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj\") pod \"success-200-isvc-a162f-predictor-5955586d98-wskzk\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") " pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.306995 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.306969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:20.423917 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.423889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:23:20.441458 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:20.441434    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:23:20.444078 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:23:20.444051    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae14de_b0a9_42da_9bc9_682dbc8bf4a0.slice/crio-1f02df0fe9ecfa3d9881c09ee8b446359e3fbdaca65f6b384f8060184f36a0ab WatchSource:0}: Error finding container 1f02df0fe9ecfa3d9881c09ee8b446359e3fbdaca65f6b384f8060184f36a0ab: Status 404 returned error can't find the container with id 1f02df0fe9ecfa3d9881c09ee8b446359e3fbdaca65f6b384f8060184f36a0ab
Aug 04 18:23:21.102977 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.102938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerStarted","Data":"360fc5cf68b25af421207478b1d2661877b6658466edf7db93eb69faf5eba534"}
Aug 04 18:23:21.103405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.103015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerStarted","Data":"bb704c5b6c80e8de701faa376b1fd9e6ea142127997dcbe61cdcca104f5b1eb9"}
Aug 04 18:23:21.103405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.103050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:21.103405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.103077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerStarted","Data":"1f02df0fe9ecfa3d9881c09ee8b446359e3fbdaca65f6b384f8060184f36a0ab"}
Aug 04 18:23:21.103405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.103105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:21.104695 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.104668    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:21.123704 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:21.123663    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podStartSLOduration=2.123648971 podStartE2EDuration="2.123648971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:21.122052106 +0000 UTC m=+1372.593339407" watchObservedRunningTime="2026-08-04 18:23:21.123648971 +0000 UTC m=+1372.594936246"
Aug 04 18:23:22.106822 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:22.106780    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:22.832946 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:22.832898    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Aug 04 18:23:23.113398 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.113364    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerID="ff1396b698144a5a28121380c7b70ccb88953c1687d45ce5603017c0f058b8be" exitCode=0
Aug 04 18:23:23.113759 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.113430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerDied","Data":"ff1396b698144a5a28121380c7b70ccb88953c1687d45ce5603017c0f058b8be"}
Aug 04 18:23:23.162685 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.162662    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:23:23.197903 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.197870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls\") pod \"d6451404-5d8d-491a-bfaf-6914ec87bb09\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") "
Aug 04 18:23:23.199999 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.199968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6451404-5d8d-491a-bfaf-6914ec87bb09" (UID: "d6451404-5d8d-491a-bfaf-6914ec87bb09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:23.301242 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.301167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config\") pod \"d6451404-5d8d-491a-bfaf-6914ec87bb09\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") "
Aug 04 18:23:23.301426 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.301292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zl8s\" (UniqueName: \"kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s\") pod \"d6451404-5d8d-491a-bfaf-6914ec87bb09\" (UID: \"d6451404-5d8d-491a-bfaf-6914ec87bb09\") "
Aug 04 18:23:23.301757 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.301731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6451404-5d8d-491a-bfaf-6914ec87bb09" (UID: "d6451404-5d8d-491a-bfaf-6914ec87bb09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:23.302043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.302028    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6451404-5d8d-491a-bfaf-6914ec87bb09-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:23.302108 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.302047    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:23.303735 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.303708    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s" (OuterVolumeSpecName: "kube-api-access-7zl8s") pod "d6451404-5d8d-491a-bfaf-6914ec87bb09" (UID: "d6451404-5d8d-491a-bfaf-6914ec87bb09"). InnerVolumeSpecName "kube-api-access-7zl8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:23.402440 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:23.402405    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zl8s\" (UniqueName: \"kubernetes.io/projected/d6451404-5d8d-491a-bfaf-6914ec87bb09-kube-api-access-7zl8s\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:23:24.118576 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.118541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg" event={"ID":"d6451404-5d8d-491a-bfaf-6914ec87bb09","Type":"ContainerDied","Data":"ec715758034f4c9c0e44f029656ca81e16f78c53e61360c870c8a4a7200ba008"}
Aug 04 18:23:24.119009 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.118612    2579 scope.go:117] "RemoveContainer" containerID="6b8f1e82c85fab45a988338f00eaf39ba1855c13181e5e294eb8e4cde4f935e9"
Aug 04 18:23:24.119009 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.118556    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"
Aug 04 18:23:24.127057 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.127036    2579 scope.go:117] "RemoveContainer" containerID="ff1396b698144a5a28121380c7b70ccb88953c1687d45ce5603017c0f058b8be"
Aug 04 18:23:24.142447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.142418    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:23:24.147236 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:24.147212    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1525-predictor-596bfc45bb-kpntg"]
Aug 04 18:23:25.030260 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:25.030217    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" path="/var/lib/kubelet/pods/d6451404-5d8d-491a-bfaf-6914ec87bb09/volumes"
Aug 04 18:23:27.112089 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:27.112060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:23:27.112559 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:27.112535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:28.074082 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:28.074042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:37.113430 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:37.113390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:38.074167 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:38.074129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:47.112876 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:47.112839    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:48.075016 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:48.074971    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Aug 04 18:23:57.113337 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:57.113298    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:23:58.075647 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:23:58.075619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:24:07.114089 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:07.114060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:24:09.821965 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:09.821936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:09.983897 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:09.983866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:19.808020 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:19.807944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.093307 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.093233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.155447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:24:20.155823 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155806    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container"
Aug 04 18:24:20.155823 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155822    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container"
Aug 04 18:24:20.155985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155831    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kube-rbac-proxy"
Aug 04 18:24:20.155985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155837    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kube-rbac-proxy"
Aug 04 18:24:20.155985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155886    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kube-rbac-proxy"
Aug 04 18:24:20.155985 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.155892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6451404-5d8d-491a-bfaf-6914ec87bb09" containerName="kserve-container"
Aug 04 18:24:20.158846 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.158823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.165826 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.165798    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d5455-predictor-serving-cert\""
Aug 04 18:24:20.165929 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.165800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d5455-kube-rbac-proxy-sar-config\""
Aug 04 18:24:20.181065 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.181038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:24:20.199394 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.199360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.292969 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.292934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.309713 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.309680    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:24:20.310255 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.310202    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container" containerID="cri-o://61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4" gracePeriod=30
Aug 04 18:24:20.310819 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.310780    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kube-rbac-proxy" containerID="cri-o://45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd" gracePeriod=30
Aug 04 18:24:20.353063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.352995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.353063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.353037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.353063 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.353056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5kn\" (UniqueName: \"kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.404196 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.404158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.454434 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.454401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.454595 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.454441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.454595 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.454458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5kn\" (UniqueName: \"kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.454595 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:24:20.454580    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d5455-predictor-serving-cert: secret "success-200-isvc-d5455-predictor-serving-cert" not found
Aug 04 18:24:20.454752 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:24:20.454662    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls podName:ee06067b-3897-497c-8c01-2f7d0a5d3962 nodeName:}" failed. No retries permitted until 2026-08-04 18:24:20.954639923 +0000 UTC m=+1432.425927192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls") pod "success-200-isvc-d5455-predictor-6776956984-nv48x" (UID: "ee06067b-3897-497c-8c01-2f7d0a5d3962") : secret "success-200-isvc-d5455-predictor-serving-cert" not found
Aug 04 18:24:20.455203 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.455165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.472899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.472867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5kn\" (UniqueName: \"kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.549909 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.549876    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.648368 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.648334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:20.957815 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.957715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:20.960081 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:20.960061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") pod \"success-200-isvc-d5455-predictor-6776956984-nv48x\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") " pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:21.069082 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.069048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:21.212428 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.212404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:24:21.215027 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:24:21.215001    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee06067b_3897_497c_8c01_2f7d0a5d3962.slice/crio-13c43d0f78febc953087a420491374feeafcc10f592758b25f0f3827c32be814 WatchSource:0}: Error finding container 13c43d0f78febc953087a420491374feeafcc10f592758b25f0f3827c32be814: Status 404 returned error can't find the container with id 13c43d0f78febc953087a420491374feeafcc10f592758b25f0f3827c32be814
Aug 04 18:24:21.336626 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.336406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerStarted","Data":"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"}
Aug 04 18:24:21.336626 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.336513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerStarted","Data":"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"}
Aug 04 18:24:21.336626 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.336553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:21.336626 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.336581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerStarted","Data":"13c43d0f78febc953087a420491374feeafcc10f592758b25f0f3827c32be814"}
Aug 04 18:24:21.338476 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.338451    2579 generic.go:356] "Generic (PLEG): container finished" podID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerID="45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd" exitCode=2
Aug 04 18:24:21.338600 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.338514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerDied","Data":"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"}
Aug 04 18:24:21.364772 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:21.364729    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podStartSLOduration=1.364715194 podStartE2EDuration="1.364715194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:21.364438731 +0000 UTC m=+1432.835726004" watchObservedRunningTime="2026-08-04 18:24:21.364715194 +0000 UTC m=+1432.836002467"
Aug 04 18:24:22.343109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:22.343076    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:22.344445 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:22.344419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:24:23.070373 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.070335    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Aug 04 18:24:23.346764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.346673    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:24:23.651210 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.651186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:24:23.780128 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.780097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") pod \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") "
Aug 04 18:24:23.780276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.780137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq56k\" (UniqueName: \"kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k\") pod \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") "
Aug 04 18:24:23.780276 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.780177    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config\") pod \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\" (UID: \"41d6b9b8-e816-4e5b-a834-9149bbfcc784\") "
Aug 04 18:24:23.780596 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.780573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41d6b9b8-e816-4e5b-a834-9149bbfcc784" (UID: "41d6b9b8-e816-4e5b-a834-9149bbfcc784"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:23.782244 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.782215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41d6b9b8-e816-4e5b-a834-9149bbfcc784" (UID: "41d6b9b8-e816-4e5b-a834-9149bbfcc784"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:23.782244 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.782226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k" (OuterVolumeSpecName: "kube-api-access-tq56k") pod "41d6b9b8-e816-4e5b-a834-9149bbfcc784" (UID: "41d6b9b8-e816-4e5b-a834-9149bbfcc784"). InnerVolumeSpecName "kube-api-access-tq56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:23.880793 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.880764    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41d6b9b8-e816-4e5b-a834-9149bbfcc784-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:24:23.880793 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.880791    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq56k\" (UniqueName: \"kubernetes.io/projected/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-api-access-tq56k\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:24:23.880977 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:23.880802    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41d6b9b8-e816-4e5b-a834-9149bbfcc784-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:24:24.351578 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.351542    2579 generic.go:356] "Generic (PLEG): container finished" podID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerID="61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4" exitCode=0
Aug 04 18:24:24.351959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.351600    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"
Aug 04 18:24:24.351959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.351631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerDied","Data":"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"}
Aug 04 18:24:24.351959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.351693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf" event={"ID":"41d6b9b8-e816-4e5b-a834-9149bbfcc784","Type":"ContainerDied","Data":"374627b1bb9fca6d6671da8ed9fe1dea1c8a56215dc61f3739aac8335a223b77"}
Aug 04 18:24:24.351959 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.351721    2579 scope.go:117] "RemoveContainer" containerID="45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"
Aug 04 18:24:24.360225 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.360207    2579 scope.go:117] "RemoveContainer" containerID="61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"
Aug 04 18:24:24.367149 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.367132    2579 scope.go:117] "RemoveContainer" containerID="45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"
Aug 04 18:24:24.367381 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:24:24.367363    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd\": container with ID starting with 45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd not found: ID does not exist" containerID="45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"
Aug 04 18:24:24.367427 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.367390    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd"} err="failed to get container status \"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd\": rpc error: code = NotFound desc = could not find container \"45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd\": container with ID starting with 45fe9f73de4e29dc4e2f9966583dccce24b7806a435f1987813ac05da39c7ddd not found: ID does not exist"
Aug 04 18:24:24.367427 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.367406    2579 scope.go:117] "RemoveContainer" containerID="61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"
Aug 04 18:24:24.367628 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:24:24.367615    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4\": container with ID starting with 61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4 not found: ID does not exist" containerID="61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"
Aug 04 18:24:24.367681 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.367630    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4"} err="failed to get container status \"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4\": rpc error: code = NotFound desc = could not find container \"61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4\": container with ID starting with 61442a8a71018872422d7ded8ad59cef9641c95158896c1eaa5c3227a35ee7c4 not found: ID does not exist"
Aug 04 18:24:24.377732 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.377714    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:24:24.380098 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:24.380079    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-94d00-predictor-85bcffb47b-j4wcf"]
Aug 04 18:24:25.030210 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:25.030179    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" path="/var/lib/kubelet/pods/41d6b9b8-e816-4e5b-a834-9149bbfcc784/volumes"
Aug 04 18:24:28.352916 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:28.352881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:24:28.353394 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:28.353368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:24:38.354323 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:38.354272    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:24:48.353668 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:48.353627    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:24:58.353745 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:24:58.353702    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Aug 04 18:25:08.354415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:08.354387    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:25:20.093792 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:20.093757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:20.774155 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:20.774111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:28.988215 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:28.988182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:25:28.988658 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:28.988219    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:25:28.994811 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:28.994788    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:25:28.995029 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:25:28.994790    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:27:07.982718 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:27:07.982680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:30:29.020293 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:30:29.020262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:30:29.020891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:30:29.020872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:30:29.026515 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:30:29.026481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:30:29.027387 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:30:29.027366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:32:34.929674 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:34.929629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:35.056304 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.056268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:32:35.056583 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.056561    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" containerID="cri-o://bb704c5b6c80e8de701faa376b1fd9e6ea142127997dcbe61cdcca104f5b1eb9" gracePeriod=30
Aug 04 18:32:35.056651 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.056592    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kube-rbac-proxy" containerID="cri-o://360fc5cf68b25af421207478b1d2661877b6658466edf7db93eb69faf5eba534" gracePeriod=30
Aug 04 18:32:35.161994 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.161960    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:32:35.162422 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162401    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kube-rbac-proxy"
Aug 04 18:32:35.162422 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162422    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kube-rbac-proxy"
Aug 04 18:32:35.162656 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162457    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container"
Aug 04 18:32:35.162656 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162466    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container"
Aug 04 18:32:35.162656 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162570    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kube-rbac-proxy"
Aug 04 18:32:35.162656 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.162583    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d6b9b8-e816-4e5b-a834-9149bbfcc784" containerName="kserve-container"
Aug 04 18:32:35.166101 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.166078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.169313 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.169290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c81a-predictor-serving-cert\""
Aug 04 18:32:35.171210 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.171193    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c81a-kube-rbac-proxy-sar-config\""
Aug 04 18:32:35.179413 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.179392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:32:35.218974 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.218906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrxt\" (UniqueName: \"kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.218974 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.218941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.219161 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.218984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.320269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.320231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrxt\" (UniqueName: \"kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.320269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.320270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.320489 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.320299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.320489 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:32:35.320408    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-serving-cert: secret "success-200-isvc-8c81a-predictor-serving-cert" not found
Aug 04 18:32:35.320489 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:32:35.320474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls podName:6aea7d0a-ac43-426f-a799-567da39ed2ad nodeName:}" failed. No retries permitted until 2026-08-04 18:32:35.820453353 +0000 UTC m=+1927.291740627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls") pod "success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" (UID: "6aea7d0a-ac43-426f-a799-567da39ed2ad") : secret "success-200-isvc-8c81a-predictor-serving-cert" not found
Aug 04 18:32:35.320944 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.320926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.334006 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.333969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrxt\" (UniqueName: \"kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.825144 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.825104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:35.827550 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:35.827510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") pod \"success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") " pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:36.078826 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:36.078737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:36.170404 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:36.170347    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerID="360fc5cf68b25af421207478b1d2661877b6658466edf7db93eb69faf5eba534" exitCode=2
Aug 04 18:32:36.170555 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:36.170400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerDied","Data":"360fc5cf68b25af421207478b1d2661877b6658466edf7db93eb69faf5eba534"}
Aug 04 18:32:36.217415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:36.217364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:32:36.220080 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:32:36.220050    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aea7d0a_ac43_426f_a799_567da39ed2ad.slice/crio-88ed44479470a810e665bc0ef0ce8f806694f1fea81cac586cf9de65fbeeb4f9 WatchSource:0}: Error finding container 88ed44479470a810e665bc0ef0ce8f806694f1fea81cac586cf9de65fbeeb4f9: Status 404 returned error can't find the container with id 88ed44479470a810e665bc0ef0ce8f806694f1fea81cac586cf9de65fbeeb4f9
Aug 04 18:32:37.107713 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.107671    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Aug 04 18:32:37.113040 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.113013    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Aug 04 18:32:37.176560 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.176522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerStarted","Data":"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"}
Aug 04 18:32:37.176726 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.176579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerStarted","Data":"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"}
Aug 04 18:32:37.176726 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.176600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerStarted","Data":"88ed44479470a810e665bc0ef0ce8f806694f1fea81cac586cf9de65fbeeb4f9"}
Aug 04 18:32:37.176726 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.176636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:37.200592 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.200546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podStartSLOduration=2.200532131 podStartE2EDuration="2.200532131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:37.19789455 +0000 UTC m=+1928.669181815" watchObservedRunningTime="2026-08-04 18:32:37.200532131 +0000 UTC m=+1928.671819405"
Aug 04 18:32:37.246468 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:37.246440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:32:38.182692 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.182654    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerID="bb704c5b6c80e8de701faa376b1fd9e6ea142127997dcbe61cdcca104f5b1eb9" exitCode=0
Aug 04 18:32:38.182692 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.182670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerDied","Data":"bb704c5b6c80e8de701faa376b1fd9e6ea142127997dcbe61cdcca104f5b1eb9"}
Aug 04 18:32:38.183081 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.183061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:38.184300 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.184274    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:32:38.199705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.199686    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:32:38.348960 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.348869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c85mj\" (UniqueName: \"kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj\") pod \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") "
Aug 04 18:32:38.348960 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.348910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config\") pod \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") "
Aug 04 18:32:38.349192 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.348979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls\") pod \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\" (UID: \"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0\") "
Aug 04 18:32:38.350983 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.349672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" (UID: "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:38.351767 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.351730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj" (OuterVolumeSpecName: "kube-api-access-c85mj") pod "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" (UID: "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0"). InnerVolumeSpecName "kube-api-access-c85mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:32:38.353990 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.353635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" (UID: "c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:32:38.450950 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.450917    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85mj\" (UniqueName: \"kubernetes.io/projected/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-api-access-c85mj\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:32:38.450950 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.450947    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:32:38.450950 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:38.450958    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:32:39.188420 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.188394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"
Aug 04 18:32:39.188877 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.188413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk" event={"ID":"c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0","Type":"ContainerDied","Data":"1f02df0fe9ecfa3d9881c09ee8b446359e3fbdaca65f6b384f8060184f36a0ab"}
Aug 04 18:32:39.188877 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.188487    2579 scope.go:117] "RemoveContainer" containerID="360fc5cf68b25af421207478b1d2661877b6658466edf7db93eb69faf5eba534"
Aug 04 18:32:39.188993 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.188919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:32:39.196530 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.196513    2579 scope.go:117] "RemoveContainer" containerID="bb704c5b6c80e8de701faa376b1fd9e6ea142127997dcbe61cdcca104f5b1eb9"
Aug 04 18:32:39.209576 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.209525    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:32:39.213659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:39.213634    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a162f-predictor-5955586d98-wskzk"]
Aug 04 18:32:41.030155 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:41.030120    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" path="/var/lib/kubelet/pods/c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0/volumes"
Aug 04 18:32:44.193509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:44.193469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:32:44.194076 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:44.194052    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:32:54.195032 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:32:54.194994    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:33:04.194534 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:04.194472    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:33:14.194002 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:14.193965    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Aug 04 18:33:24.195174 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:24.195092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:33:34.854138 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:34.854091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:34.955420 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:34.955386    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:33:34.955677 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:34.955650    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container" containerID="cri-o://3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3" gracePeriod=30
Aug 04 18:33:34.955769 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:34.955691    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kube-rbac-proxy" containerID="cri-o://5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d" gracePeriod=30
Aug 04 18:33:34.981800 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:34.981772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.040244 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040213    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:33:35.040523 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container"
Aug 04 18:33:35.040523 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040523    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container"
Aug 04 18:33:35.040764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kube-rbac-proxy"
Aug 04 18:33:35.040764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040547    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kube-rbac-proxy"
Aug 04 18:33:35.040764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040609    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kube-rbac-proxy"
Aug 04 18:33:35.040764 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.040619    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ae14de-b0a9-42da-9bc9-682dbc8bf4a0" containerName="kserve-container"
Aug 04 18:33:35.043604 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.043585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.046241 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.046218    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-493ba-kube-rbac-proxy-sar-config\""
Aug 04 18:33:35.046241 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.046236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-493ba-predictor-serving-cert\""
Aug 04 18:33:35.054051 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.054030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:33:35.060666 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.060641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.071453 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.071431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.071583 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.071469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdv7f\" (UniqueName: \"kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.071671 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.071582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.142394 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.142346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.172222 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.172194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.172395 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.172238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.172395 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.172264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdv7f\" (UniqueName: \"kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.172493 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:35.172410    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-493ba-predictor-serving-cert: secret "success-200-isvc-493ba-predictor-serving-cert" not found
Aug 04 18:33:35.172493 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:35.172477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls podName:d70681f7-5e81-4c3b-8b8d-3d965875b8a0 nodeName:}" failed. No retries permitted until 2026-08-04 18:33:35.672461819 +0000 UTC m=+1987.143749074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls") pod "success-200-isvc-493ba-predictor-587766b78d-tbfg9" (UID: "d70681f7-5e81-4c3b-8b8d-3d965875b8a0") : secret "success-200-isvc-493ba-predictor-serving-cert" not found
Aug 04 18:33:35.172891 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.172872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.185034 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.185008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdv7f\" (UniqueName: \"kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.230118 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.230077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.329951 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.329904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.395249 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.395165    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerID="5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d" exitCode=2
Aug 04 18:33:35.395249 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.395239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerDied","Data":"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"}
Aug 04 18:33:35.431436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.431406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:35.676912 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.676817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.679401 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.679368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") pod \"success-200-isvc-493ba-predictor-587766b78d-tbfg9\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") " pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:35.955205 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:35.955102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:36.088043 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.088011    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:33:36.090612 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:33:36.090587    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70681f7_5e81_4c3b_8b8d_3d965875b8a0.slice/crio-e22e694cf04afc3c983f9d483e3b5442076c7900b193a9ccb2c262089e209c17 WatchSource:0}: Error finding container e22e694cf04afc3c983f9d483e3b5442076c7900b193a9ccb2c262089e209c17: Status 404 returned error can't find the container with id e22e694cf04afc3c983f9d483e3b5442076c7900b193a9ccb2c262089e209c17
Aug 04 18:33:36.401145 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.401102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerStarted","Data":"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"}
Aug 04 18:33:36.401317 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.401162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerStarted","Data":"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"}
Aug 04 18:33:36.401317 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.401188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerStarted","Data":"e22e694cf04afc3c983f9d483e3b5442076c7900b193a9ccb2c262089e209c17"}
Aug 04 18:33:36.401317 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.401213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:36.401317 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.401252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:36.402838 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.402812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:33:36.423377 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:36.423327    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podStartSLOduration=1.4233114439999999 podStartE2EDuration="1.423311444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:36.420990772 +0000 UTC m=+1987.892278070" watchObservedRunningTime="2026-08-04 18:33:36.423311444 +0000 UTC m=+1987.894598718"
Aug 04 18:33:37.404469 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:37.404432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:33:38.103827 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.103800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:33:38.200155 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.200065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m5kn\" (UniqueName: \"kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn\") pod \"ee06067b-3897-497c-8c01-2f7d0a5d3962\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") "
Aug 04 18:33:38.200155 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.200137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") pod \"ee06067b-3897-497c-8c01-2f7d0a5d3962\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") "
Aug 04 18:33:38.200376 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.200188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config\") pod \"ee06067b-3897-497c-8c01-2f7d0a5d3962\" (UID: \"ee06067b-3897-497c-8c01-2f7d0a5d3962\") "
Aug 04 18:33:38.200589 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.200554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee06067b-3897-497c-8c01-2f7d0a5d3962" (UID: "ee06067b-3897-497c-8c01-2f7d0a5d3962"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:33:38.202202 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.202169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee06067b-3897-497c-8c01-2f7d0a5d3962" (UID: "ee06067b-3897-497c-8c01-2f7d0a5d3962"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:33:38.202288 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.202207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn" (OuterVolumeSpecName: "kube-api-access-8m5kn") pod "ee06067b-3897-497c-8c01-2f7d0a5d3962" (UID: "ee06067b-3897-497c-8c01-2f7d0a5d3962"). InnerVolumeSpecName "kube-api-access-8m5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:33:38.301597 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.301556    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5kn\" (UniqueName: \"kubernetes.io/projected/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-api-access-8m5kn\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:38.301597 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.301586    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee06067b-3897-497c-8c01-2f7d0a5d3962-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:38.301597 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.301598    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee06067b-3897-497c-8c01-2f7d0a5d3962-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:38.409966 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.409922    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerID="3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3" exitCode=0
Aug 04 18:33:38.409966 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.409959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerDied","Data":"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"}
Aug 04 18:33:38.409966 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.409971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"
Aug 04 18:33:38.410447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.410009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x" event={"ID":"ee06067b-3897-497c-8c01-2f7d0a5d3962","Type":"ContainerDied","Data":"13c43d0f78febc953087a420491374feeafcc10f592758b25f0f3827c32be814"}
Aug 04 18:33:38.410447 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.410057    2579 scope.go:117] "RemoveContainer" containerID="5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"
Aug 04 18:33:38.418944 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.418863    2579 scope.go:117] "RemoveContainer" containerID="3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"
Aug 04 18:33:38.425899 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.425880    2579 scope.go:117] "RemoveContainer" containerID="5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"
Aug 04 18:33:38.426156 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:38.426136    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d\": container with ID starting with 5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d not found: ID does not exist" containerID="5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"
Aug 04 18:33:38.426203 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.426165    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d"} err="failed to get container status \"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d\": rpc error: code = NotFound desc = could not find container \"5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d\": container with ID starting with 5f24abda3ddc4f05f62c68170d8b9ac74c6706dd34d6aad404e1f3c261c00f9d not found: ID does not exist"
Aug 04 18:33:38.426203 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.426182    2579 scope.go:117] "RemoveContainer" containerID="3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"
Aug 04 18:33:38.426450 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:38.426431    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3\": container with ID starting with 3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3 not found: ID does not exist" containerID="3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"
Aug 04 18:33:38.426524 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.426456    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3"} err="failed to get container status \"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3\": rpc error: code = NotFound desc = could not find container \"3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3\": container with ID starting with 3e8cd26c773cb21edc323bf4765ce7004ec56a2a922693d1ec319e2e554d00d3 not found: ID does not exist"
Aug 04 18:33:38.435689 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.435665    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:33:38.440062 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:38.440045    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d5455-predictor-6776956984-nv48x"]
Aug 04 18:33:39.035705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:39.035662    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" path="/var/lib/kubelet/pods/ee06067b-3897-497c-8c01-2f7d0a5d3962/volumes"
Aug 04 18:33:42.412729 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:42.412702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:33:42.413220 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:42.413190    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:33:45.235913 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.235868    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:45.333688 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.333640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:45.352167 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.352133    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:33:45.352465 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.352438    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container" containerID="cri-o://c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2" gracePeriod=30
Aug 04 18:33:45.352567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.352468    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kube-rbac-proxy" containerID="cri-o://97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569" gracePeriod=30
Aug 04 18:33:45.391262 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391223    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:33:45.391567 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391554    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kube-rbac-proxy"
Aug 04 18:33:45.391623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391570    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kube-rbac-proxy"
Aug 04 18:33:45.391623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391589    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container"
Aug 04 18:33:45.391623 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container"
Aug 04 18:33:45.391740 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391652    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kube-rbac-proxy"
Aug 04 18:33:45.391740 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.391661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee06067b-3897-497c-8c01-2f7d0a5d3962" containerName="kserve-container"
Aug 04 18:33:45.396016 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.395999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.398165 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.398141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-47ede-predictor-serving-cert\""
Aug 04 18:33:45.398250 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.398176    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-47ede-kube-rbac-proxy-sar-config\""
Aug 04 18:33:45.400314 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.400291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:45.404836 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.404814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:33:45.461410 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.461388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.461571 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.461420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlc2\" (UniqueName: \"kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.461571 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.461444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.486956 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.486894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:45.561961 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.561928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlc2\" (UniqueName: \"kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.562109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.561979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.562109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.562061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.562230 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:45.562139    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-47ede-predictor-serving-cert: secret "success-200-isvc-47ede-predictor-serving-cert" not found
Aug 04 18:33:45.562230 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:45.562223    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls podName:a28fd857-4602-4468-9285-e3a787043a5e nodeName:}" failed. No retries permitted until 2026-08-04 18:33:46.062194016 +0000 UTC m=+1997.533481272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls") pod "success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" (UID: "a28fd857-4602-4468-9285-e3a787043a5e") : secret "success-200-isvc-47ede-predictor-serving-cert" not found
Aug 04 18:33:45.562712 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.562690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.572863 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.572844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlc2\" (UniqueName: \"kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:45.646398 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.646354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:45.746142 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:45.746066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:33:46.065730 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.065633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:46.068062 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.068040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") pod \"success-200-isvc-47ede-predictor-6fd67659b4-bxjfg\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") " pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:46.306512 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.306462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:46.437898 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.437801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:33:46.440864 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:33:46.440835    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28fd857_4602_4468_9285_e3a787043a5e.slice/crio-d2d3e65eef17731a40a6fd62e7f17749e58cf34d78832d8e173b8266fac9d741 WatchSource:0}: Error finding container d2d3e65eef17731a40a6fd62e7f17749e58cf34d78832d8e173b8266fac9d741: Status 404 returned error can't find the container with id d2d3e65eef17731a40a6fd62e7f17749e58cf34d78832d8e173b8266fac9d741
Aug 04 18:33:46.445269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.445247    2579 generic.go:356] "Generic (PLEG): container finished" podID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerID="97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569" exitCode=2
Aug 04 18:33:46.445363 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:46.445313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerDied","Data":"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"}
Aug 04 18:33:47.450260 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:47.450214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerStarted","Data":"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"}
Aug 04 18:33:47.450659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:47.450291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerStarted","Data":"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"}
Aug 04 18:33:47.450659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:47.450328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:47.450659 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:47.450355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerStarted","Data":"d2d3e65eef17731a40a6fd62e7f17749e58cf34d78832d8e173b8266fac9d741"}
Aug 04 18:33:47.472111 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:47.472065    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podStartSLOduration=2.4720515069999998 podStartE2EDuration="2.472051507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:47.470723607 +0000 UTC m=+1998.942010882" watchObservedRunningTime="2026-08-04 18:33:47.472051507 +0000 UTC m=+1998.943338781"
Aug 04 18:33:48.393386 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.393365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:33:48.455171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455141    2579 generic.go:356] "Generic (PLEG): container finished" podID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerID="c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2" exitCode=0
Aug 04 18:33:48.455528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455188    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"
Aug 04 18:33:48.455528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerDied","Data":"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"}
Aug 04 18:33:48.455528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg" event={"ID":"6aea7d0a-ac43-426f-a799-567da39ed2ad","Type":"ContainerDied","Data":"88ed44479470a810e665bc0ef0ce8f806694f1fea81cac586cf9de65fbeeb4f9"}
Aug 04 18:33:48.455528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455310    2579 scope.go:117] "RemoveContainer" containerID="97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"
Aug 04 18:33:48.455528 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.455525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:48.456831 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.456797    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:33:48.463183 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.463168    2579 scope.go:117] "RemoveContainer" containerID="c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"
Aug 04 18:33:48.470016 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.470001    2579 scope.go:117] "RemoveContainer" containerID="97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"
Aug 04 18:33:48.470249 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:48.470231    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569\": container with ID starting with 97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569 not found: ID does not exist" containerID="97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"
Aug 04 18:33:48.470299 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.470259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569"} err="failed to get container status \"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569\": rpc error: code = NotFound desc = could not find container \"97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569\": container with ID starting with 97289675650cb257d6de828858ecb64f423ec8f7e59819262fff2af1b2dee569 not found: ID does not exist"
Aug 04 18:33:48.470299 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.470274    2579 scope.go:117] "RemoveContainer" containerID="c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"
Aug 04 18:33:48.470535 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:33:48.470518    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2\": container with ID starting with c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2 not found: ID does not exist" containerID="c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"
Aug 04 18:33:48.470580 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.470541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2"} err="failed to get container status \"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2\": rpc error: code = NotFound desc = could not find container \"c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2\": container with ID starting with c247b917669eedb1c588a78b3deb58ff7c4f9f4e4623925e820a20340f66ada2 not found: ID does not exist"
Aug 04 18:33:48.485864 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.485826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkrxt\" (UniqueName: \"kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt\") pod \"6aea7d0a-ac43-426f-a799-567da39ed2ad\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") "
Aug 04 18:33:48.485920 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.485870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config\") pod \"6aea7d0a-ac43-426f-a799-567da39ed2ad\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") "
Aug 04 18:33:48.485920 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.485899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") pod \"6aea7d0a-ac43-426f-a799-567da39ed2ad\" (UID: \"6aea7d0a-ac43-426f-a799-567da39ed2ad\") "
Aug 04 18:33:48.486218 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.486197    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6aea7d0a-ac43-426f-a799-567da39ed2ad" (UID: "6aea7d0a-ac43-426f-a799-567da39ed2ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:33:48.487785 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.487767    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt" (OuterVolumeSpecName: "kube-api-access-pkrxt") pod "6aea7d0a-ac43-426f-a799-567da39ed2ad" (UID: "6aea7d0a-ac43-426f-a799-567da39ed2ad"). InnerVolumeSpecName "kube-api-access-pkrxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:33:48.487851 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.487808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6aea7d0a-ac43-426f-a799-567da39ed2ad" (UID: "6aea7d0a-ac43-426f-a799-567da39ed2ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:33:48.587286 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.587263    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrxt\" (UniqueName: \"kubernetes.io/projected/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-api-access-pkrxt\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:48.587286 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.587284    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aea7d0a-ac43-426f-a799-567da39ed2ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:48.587455 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.587294    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aea7d0a-ac43-426f-a799-567da39ed2ad-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:33:48.780869 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.780842    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:33:48.790917 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:48.787177    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c81a-predictor-f8d8945c6-v2fbg"]
Aug 04 18:33:49.030038 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:49.029955    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" path="/var/lib/kubelet/pods/6aea7d0a-ac43-426f-a799-567da39ed2ad/volumes"
Aug 04 18:33:49.460685 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:49.460647    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:33:52.413407 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:52.413368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:33:54.465308 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:54.465279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:33:54.465698 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:33:54.465677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:34:02.414086 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:02.414045    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:34:04.465770 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:04.465734    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:34:12.413783 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:12.413744    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Aug 04 18:34:13.063767 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:13.063730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:14.466670 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:14.466625    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:34:22.413660 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:22.413630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:34:24.465780 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:24.465733    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Aug 04 18:34:31.735344 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:31.735303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:32.453643 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:32.453607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:34.467435 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:34.467403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:34:35.115968 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:35.115937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:35.378808 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:35.378727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:45.525009 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:45.524961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:34:45.743347 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:34:45.743305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:17.515116 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:35:17.515028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:35:29.042634 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:35:29.042610    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:35:29.045047 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:35:29.044205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:35:29.048713 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:35:29.048688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:35:29.050260 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:35:29.050234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:39:37.331476 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:39:37.331399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:40:29.069100 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:40:29.068998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:40:29.072907 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:40:29.070696    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:40:29.077680 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:40:29.077658    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:40:29.079036 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:40:29.079018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:41:25.697355 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:41:25.697270    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:00.237254 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.237211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:00.421981 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.421950    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:43:00.422289 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.422259    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container" containerID="cri-o://89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30" gracePeriod=30
Aug 04 18:43:00.422405 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.422298    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kube-rbac-proxy" containerID="cri-o://ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5" gracePeriod=30
Aug 04 18:43:00.459881 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.459853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:00.536316 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:00.536241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:43:01.476109 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:01.476075    2579 generic.go:356] "Generic (PLEG): container finished" podID="a28fd857-4602-4468-9285-e3a787043a5e" containerID="ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5" exitCode=2
Aug 04 18:43:01.476471 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:01.476146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerDied","Data":"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"}
Aug 04 18:43:03.264391 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.264369    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:43:03.411206 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.411171    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdlc2\" (UniqueName: \"kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2\") pod \"a28fd857-4602-4468-9285-e3a787043a5e\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") "
Aug 04 18:43:03.411361 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.411228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config\") pod \"a28fd857-4602-4468-9285-e3a787043a5e\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") "
Aug 04 18:43:03.411361 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.411255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") pod \"a28fd857-4602-4468-9285-e3a787043a5e\" (UID: \"a28fd857-4602-4468-9285-e3a787043a5e\") "
Aug 04 18:43:03.411636 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.411611    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a28fd857-4602-4468-9285-e3a787043a5e" (UID: "a28fd857-4602-4468-9285-e3a787043a5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:43:03.413345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.413311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2" (OuterVolumeSpecName: "kube-api-access-tdlc2") pod "a28fd857-4602-4468-9285-e3a787043a5e" (UID: "a28fd857-4602-4468-9285-e3a787043a5e"). InnerVolumeSpecName "kube-api-access-tdlc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:43:03.413345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.413326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a28fd857-4602-4468-9285-e3a787043a5e" (UID: "a28fd857-4602-4468-9285-e3a787043a5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:43:03.484842 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.484810    2579 generic.go:356] "Generic (PLEG): container finished" podID="a28fd857-4602-4468-9285-e3a787043a5e" containerID="89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30" exitCode=0
Aug 04 18:43:03.484970 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.484859    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"
Aug 04 18:43:03.484970 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.484861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerDied","Data":"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"}
Aug 04 18:43:03.485040 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.484977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg" event={"ID":"a28fd857-4602-4468-9285-e3a787043a5e","Type":"ContainerDied","Data":"d2d3e65eef17731a40a6fd62e7f17749e58cf34d78832d8e173b8266fac9d741"}
Aug 04 18:43:03.485040 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.485007    2579 scope.go:117] "RemoveContainer" containerID="ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"
Aug 04 18:43:03.493568 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.493550    2579 scope.go:117] "RemoveContainer" containerID="89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"
Aug 04 18:43:03.500385 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.500367    2579 scope.go:117] "RemoveContainer" containerID="ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"
Aug 04 18:43:03.500667 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:43:03.500649    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5\": container with ID starting with ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5 not found: ID does not exist" containerID="ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"
Aug 04 18:43:03.500730 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.500674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5"} err="failed to get container status \"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5\": rpc error: code = NotFound desc = could not find container \"ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5\": container with ID starting with ffdf225cf4134ea0a3b5a252213ad90eb2c6bbb4a7a50ab898afe35aa35180c5 not found: ID does not exist"
Aug 04 18:43:03.500730 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.500691    2579 scope.go:117] "RemoveContainer" containerID="89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"
Aug 04 18:43:03.500924 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:43:03.500907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30\": container with ID starting with 89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30 not found: ID does not exist" containerID="89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"
Aug 04 18:43:03.500971 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.500930    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30"} err="failed to get container status \"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30\": rpc error: code = NotFound desc = could not find container \"89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30\": container with ID starting with 89a38c009b62f1be6ce72a3b89a452ced41155898a66045bcbc2f1b787e6dc30 not found: ID does not exist"
Aug 04 18:43:03.509560 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.509486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:43:03.512359 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.512342    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlc2\" (UniqueName: \"kubernetes.io/projected/a28fd857-4602-4468-9285-e3a787043a5e-kube-api-access-tdlc2\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:43:03.512415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.512366    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a28fd857-4602-4468-9285-e3a787043a5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:43:03.512415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.512380    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a28fd857-4602-4468-9285-e3a787043a5e-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:43:03.514989 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:03.514969    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-47ede-predictor-6fd67659b4-bxjfg"]
Aug 04 18:43:05.030171 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:05.030143    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a28fd857-4602-4468-9285-e3a787043a5e" path="/var/lib/kubelet/pods/a28fd857-4602-4468-9285-e3a787043a5e/volumes"
Aug 04 18:43:49.076931 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:43:49.076896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:45:29.095837 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:45:29.095724    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:45:29.099990 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:45:29.097682    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:45:29.101867 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:45:29.101848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:45:29.103635 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:45:29.103616    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:50:29.118775 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:29.118746    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:50:29.121196 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:29.121175    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:50:29.125535 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:29.125511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:50:29.127866 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:29.127848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:50:54.408930 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:54.408900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:54.545359 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:54.545330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:50:54.545616 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:54.545586    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container" containerID="cri-o://a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5" gracePeriod=30
Aug 04 18:50:54.545740 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:54.545615    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kube-rbac-proxy" containerID="cri-o://431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827" gracePeriod=30
Aug 04 18:50:55.211987 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:55.211954    2579 generic.go:356] "Generic (PLEG): container finished" podID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerID="431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827" exitCode=2
Aug 04 18:50:55.212149 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:55.212013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerDied","Data":"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"}
Aug 04 18:50:56.403976 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:56.403942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:50:57.075725 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.075702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:50:57.205345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.205277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") pod \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") "
Aug 04 18:50:57.205345 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.205338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config\") pod \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") "
Aug 04 18:50:57.205551 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.205385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdv7f\" (UniqueName: \"kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f\") pod \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\" (UID: \"d70681f7-5e81-4c3b-8b8d-3d965875b8a0\") "
Aug 04 18:50:57.205714 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.205685    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d70681f7-5e81-4c3b-8b8d-3d965875b8a0" (UID: "d70681f7-5e81-4c3b-8b8d-3d965875b8a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:50:57.207080 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.207062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d70681f7-5e81-4c3b-8b8d-3d965875b8a0" (UID: "d70681f7-5e81-4c3b-8b8d-3d965875b8a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:50:57.207197 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.207175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f" (OuterVolumeSpecName: "kube-api-access-pdv7f") pod "d70681f7-5e81-4c3b-8b8d-3d965875b8a0" (UID: "d70681f7-5e81-4c3b-8b8d-3d965875b8a0"). InnerVolumeSpecName "kube-api-access-pdv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:50:57.220415 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.220393    2579 generic.go:356] "Generic (PLEG): container finished" podID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerID="a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5" exitCode=0
Aug 04 18:50:57.220509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.220431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerDied","Data":"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"}
Aug 04 18:50:57.220509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.220439    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"
Aug 04 18:50:57.220509 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.220475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9" event={"ID":"d70681f7-5e81-4c3b-8b8d-3d965875b8a0","Type":"ContainerDied","Data":"e22e694cf04afc3c983f9d483e3b5442076c7900b193a9ccb2c262089e209c17"}
Aug 04 18:50:57.220625 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.220522    2579 scope.go:117] "RemoveContainer" containerID="431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"
Aug 04 18:50:57.228230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.228063    2579 scope.go:117] "RemoveContainer" containerID="a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"
Aug 04 18:50:57.234814 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.234799    2579 scope.go:117] "RemoveContainer" containerID="431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"
Aug 04 18:50:57.235047 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:50:57.235031    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827\": container with ID starting with 431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827 not found: ID does not exist" containerID="431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"
Aug 04 18:50:57.235124 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.235057    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827"} err="failed to get container status \"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827\": rpc error: code = NotFound desc = could not find container \"431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827\": container with ID starting with 431b55f3a0c374644b3e064d37e65b6d29261d45bede8551bab02747459f7827 not found: ID does not exist"
Aug 04 18:50:57.235124 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.235076    2579 scope.go:117] "RemoveContainer" containerID="a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"
Aug 04 18:50:57.235305 ip-10-0-133-250 kubenswrapper[2579]: E0804 18:50:57.235288    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5\": container with ID starting with a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5 not found: ID does not exist" containerID="a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"
Aug 04 18:50:57.235343 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.235311    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5"} err="failed to get container status \"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5\": rpc error: code = NotFound desc = could not find container \"a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5\": container with ID starting with a0ba5053b718923ca813adbb51a02d349fcc0a5e981cd4f3101d3be10ce6aec5 not found: ID does not exist"
Aug 04 18:50:57.244106 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.244084    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:50:57.249006 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.248986    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-493ba-predictor-587766b78d-tbfg9"]
Aug 04 18:50:57.306703 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.306682    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-proxy-tls\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:50:57.306703 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.306700    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:50:57.306823 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:57.306715    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdv7f\" (UniqueName: \"kubernetes.io/projected/d70681f7-5e81-4c3b-8b8d-3d965875b8a0-kube-api-access-pdv7f\") on node \"ip-10-0-133-250.ec2.internal\" DevicePath \"\""
Aug 04 18:50:59.029005 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:50:59.028974    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" path="/var/lib/kubelet/pods/d70681f7-5e81-4c3b-8b8d-3d965875b8a0/volumes"
Aug 04 18:51:11.663072 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:11.663037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:27.457432 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:27.457349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:27.640611 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:27.640579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:27.702122 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:27.702090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:30.288116 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:30.288088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m5ldx_7d06c764-6c99-4554-a19c-b8f08450baef/global-pull-secret-syncer/0.log"
Aug 04 18:51:30.353624 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:30.353602    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log"
Aug 04 18:51:30.507214 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:30.507179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-250.ec2.internal_d77a65267e4463f8e5a16dfb51a44592/haproxy/0.log"
Aug 04 18:51:33.493916 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:33.493883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:33.559338 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:33.559307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9fwpg_ed50106d-71e8-4606-a3fd-081ac58e180c/konnectivity-agent/0.log" limit=1024
Aug 04 18:51:34.785256 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:34.785178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-n562v_8ef40055-92c5-45a1-ad48-9f086f1292b2/cluster-monitoring-operator/0.log"
Aug 04 18:51:35.009048 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:35.009024    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47pcn_4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d/node-exporter/0.log"
Aug 04 18:51:35.041250 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:35.041176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47pcn_4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d/kube-rbac-proxy/0.log"
Aug 04 18:51:35.070386 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:35.070364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-47pcn_4fcf0a2e-be82-4ed5-89d8-2665f0ccca8d/init-textfile/0.log"
Aug 04 18:51:37.291013 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:37.290986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-zdzfm_443df73d-2731-4d1c-8825-22a0b24b0a9c/networking-console-plugin/0.log"
Aug 04 18:51:37.812182 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:37.812150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/1.log"
Aug 04 18:51:37.816600 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:37.816581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-pglq2_d6a9867b-b02f-4f60-8a12-783faccd03cd/console-operator/2.log"
Aug 04 18:51:39.838656 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.838629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbmqg_008642e1-a70d-496a-876c-e70fc8b02be0/dns/0.log"
Aug 04 18:51:39.865431 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.865410    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bbmqg_008642e1-a70d-496a-876c-e70fc8b02be0/kube-rbac-proxy/0.log"
Aug 04 18:51:39.962825 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.962800    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"]
Aug 04 18:51:39.963047 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963048    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963059    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963064    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963070    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963077    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963088 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963087    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963093    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963124    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963128    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963184    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aea7d0a-ac43-426f-a799-567da39ed2ad" containerName="kube-rbac-proxy"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70681f7-5e81-4c3b-8b8d-3d965875b8a0" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kserve-container"
Aug 04 18:51:39.963280 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.963202    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a28fd857-4602-4468-9285-e3a787043a5e" containerName="kube-rbac-proxy"
Aug 04 18:51:39.967227 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.967213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:39.973212 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.973189    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh2h8\"/\"kube-root-ca.crt\""
Aug 04 18:51:39.973332 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.973189    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bh2h8\"/\"openshift-service-ca.crt\""
Aug 04 18:51:39.973332 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.973246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bh2h8\"/\"default-dockercfg-7nj45\""
Aug 04 18:51:39.975682 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.975664    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8vf5_21136c2d-8c4a-41d9-9b45-9d55c3599539/dns-node-resolver/0.log"
Aug 04 18:51:39.989788 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:39.989759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"]
Aug 04 18:51:40.121687 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.121659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-proc\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.121798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.121708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-podres\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.121798 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.121768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnzz\" (UniqueName: \"kubernetes.io/projected/89af3575-608a-4e59-8bec-4b96f5982149-kube-api-access-fwnzz\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.121876 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.121810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-sys\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.121876 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.121839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-lib-modules\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223235 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-proc\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223235 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-podres\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnzz\" (UniqueName: \"kubernetes.io/projected/89af3575-608a-4e59-8bec-4b96f5982149-kube-api-access-fwnzz\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-proc\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-podres\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223436 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-sys\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223630 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-lib-modules\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223630 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-sys\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.223630 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.223585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89af3575-608a-4e59-8bec-4b96f5982149-lib-modules\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.232640 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.232625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnzz\" (UniqueName: \"kubernetes.io/projected/89af3575-608a-4e59-8bec-4b96f5982149-kube-api-access-fwnzz\") pod \"perf-node-gather-daemonset-wv7wf\" (UID: \"89af3575-608a-4e59-8bec-4b96f5982149\") " pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.276230 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.276205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:40.399228 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.399154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"]
Aug 04 18:51:40.402720 ip-10-0-133-250 kubenswrapper[2579]: W0804 18:51:40.402692    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod89af3575_608a_4e59_8bec_4b96f5982149.slice/crio-277a6861d5281151e598a5c5965e21c2a62ae2bd4811a5178a476240372ffdc9 WatchSource:0}: Error finding container 277a6861d5281151e598a5c5965e21c2a62ae2bd4811a5178a476240372ffdc9: Status 404 returned error can't find the container with id 277a6861d5281151e598a5c5965e21c2a62ae2bd4811a5178a476240372ffdc9
Aug 04 18:51:40.599849 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:40.599826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d8j6c_5082e209-4074-4742-9da7-0ce1bb74779f/node-ca/0.log"
Aug 04 18:51:41.377638 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.377608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf" event={"ID":"89af3575-608a-4e59-8bec-4b96f5982149","Type":"ContainerStarted","Data":"020e305aa7bacefb5e2727480f67297bd101c015b93bef0d4960a8a326a0aea4"}
Aug 04 18:51:41.378098 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.377658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf" event={"ID":"89af3575-608a-4e59-8bec-4b96f5982149","Type":"ContainerStarted","Data":"277a6861d5281151e598a5c5965e21c2a62ae2bd4811a5178a476240372ffdc9"}
Aug 04 18:51:41.378098 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.377697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:41.399529 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.399470    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf" podStartSLOduration=2.399456789 podStartE2EDuration="2.399456789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:51:41.398288077 +0000 UTC m=+3072.869575349" watchObservedRunningTime="2026-08-04 18:51:41.399456789 +0000 UTC m=+3072.870744061"
Aug 04 18:51:41.446443 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.446417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f6d8894dd-p9sxx_c61d8d4e-6d24-4d27-88d3-bb909065ede6/router/0.log"
Aug 04 18:51:41.825278 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:41.825192    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6dvr_d39c81e0-45c6-4b3e-beb7-d9795902a1e8/serve-healthcheck-canary/0.log"
Aug 04 18:51:42.279973 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:42.279939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wwdm7_693fec9c-5ef1-47ed-97d5-52b6d6c8b16b/insights-operator/0.log"
Aug 04 18:51:42.280757 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:42.280740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wwdm7_693fec9c-5ef1-47ed-97d5-52b6d6c8b16b/insights-operator/1.log"
Aug 04 18:51:42.468479 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:42.468448    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj6wg_59658784-188e-4a4e-aa98-1e9089818eee/kube-rbac-proxy/0.log"
Aug 04 18:51:42.493002 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:42.492972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj6wg_59658784-188e-4a4e-aa98-1e9089818eee/exporter/0.log"
Aug 04 18:51:42.518921 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:42.518887    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vj6wg_59658784-188e-4a4e-aa98-1e9089818eee/extractor/0.log"
Aug 04 18:51:44.835130 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:44.835076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5f9d7f69b6-x76v9_46c28850-2cc4-4c05-a90c-9690a485ea15/manager/0.log"
Aug 04 18:51:44.897216 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:44.897189    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-xr8g9_5dd6d4cc-5d45-4ba6-b2ed-241fbdae8902/server/0.log"
Aug 04 18:51:47.390248 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:47.390222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bh2h8/perf-node-gather-daemonset-wv7wf"
Aug 04 18:51:51.185024 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.184960    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4nbss_6cbd87d7-6707-4dd7-aa6b-2d3a8fe85a8e/kube-multus/0.log"
Aug 04 18:51:51.217249 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.217227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/kube-multus-additional-cni-plugins/0.log"
Aug 04 18:51:51.248211 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.248194    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/egress-router-binary-copy/0.log"
Aug 04 18:51:51.289592 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.289576    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/cni-plugins/0.log"
Aug 04 18:51:51.323796 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.323774    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/bond-cni-plugin/0.log"
Aug 04 18:51:51.355090 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.355073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/routeoverride-cni/0.log"
Aug 04 18:51:51.383344 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.383328    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/whereabouts-cni-bincopy/0.log"
Aug 04 18:51:51.410503 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:51.410481    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4ktf9_6642e299-8f13-49a5-835f-02c797ab231e/whereabouts-cni/0.log"
Aug 04 18:51:52.007313 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:52.007282    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw5pt_2550b4e7-088e-48e4-abc9-2a500311fabc/network-metrics-daemon/0.log"
Aug 04 18:51:52.031705 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:52.031639    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jw5pt_2550b4e7-088e-48e4-abc9-2a500311fabc/kube-rbac-proxy/0.log"
Aug 04 18:51:53.179882 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.179853    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-controller/0.log"
Aug 04 18:51:53.206459 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.206439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/0.log"
Aug 04 18:51:53.206612 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.206553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovn-acl-logging/1.log"
Aug 04 18:51:53.228815 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.228796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/kube-rbac-proxy-node/0.log"
Aug 04 18:51:53.255475 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.255455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/kube-rbac-proxy-ovn-metrics/0.log"
Aug 04 18:51:53.281981 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.281962    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/northd/0.log"
Aug 04 18:51:53.308269 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.308253    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/nbdb/0.log"
Aug 04 18:51:53.336352 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.336326    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/sbdb/0.log"
Aug 04 18:51:53.382566 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:53.382548    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dwcm5_8f0ebca0-e0b5-47d3-8f9e-0ec3a22e1353/ovnkube-controller/0.log"
Aug 04 18:51:54.781302 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:54.781272    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tglfd_56060bb1-3104-4bbb-beb9-008c28976663/network-check-target-container/0.log"
Aug 04 18:51:55.858820 ip-10-0-133-250 kubenswrapper[2579]: I0804 18:51:55.858745    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ktn7p_d9722643-4aff-48fd-8bef-8abe70c56b18/iptables-alerter/0.log"