Jul 09 18:02:13.267189 ip-10-0-138-20 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 18:02:13.267202 ip-10-0-138-20 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 18:02:13.267209 ip-10-0-138-20 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 18:02:13.267453 ip-10-0-138-20 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 18:02:23.473172 ip-10-0-138-20 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 18:02:23.473189 ip-10-0-138-20 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f4b12322a57b440fa25f18a55d657e0a --
Jul 09 18:04:27.144768 ip-10-0-138-20 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:04:29.182565 ip-10-0-138-20 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:29.182565 ip-10-0-138-20 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:04:29.182565 ip-10-0-138-20 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:29.182565 ip-10-0-138-20 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:04:29.182565 ip-10-0-138-20 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:04:29.185234 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.185084    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:04:29.188760 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188743    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.188760 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188759    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188763    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188767    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188770    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188773    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188776    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188778    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188781    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188783    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188786    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188789    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188792    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188795    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188798    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188801    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188803    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188806    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188809    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188816    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188819    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.188825 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188822    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188825    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188828    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188831    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188834    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188837    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188840    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188843    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188845    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188848    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188850    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188853    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188855    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188858    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188860    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188862    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188865    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188867    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188870    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188872    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.189289 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188875    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188880    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188884    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188887    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188889    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188892    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188894    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188897    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188899    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188902    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188904    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188907    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188909    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188912    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188915    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188918    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188920    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188923    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188926    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.189809 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188928    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188930    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188933    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188935    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188938    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188940    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188943    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188946    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188948    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188950    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188953    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188956    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188958    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188961    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188963    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188966    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188968    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188970    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188973    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188975    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.190257 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188978    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188980    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188984    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188987    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188990    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188993    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.188998    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189002    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189005    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189007    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189010    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189014    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189018    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189021    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189024    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189027    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189030    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189032    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189035    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.190756 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189788    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189795    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189797    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189800    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189803    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189806    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189809    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189812    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189815    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189818    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189822    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189826    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189829    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189832    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189835    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189837    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189840    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189843    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189845    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.191240 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189848    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189852    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189855    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189858    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189862    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189865    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189869    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189871    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189874    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189877    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189879    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189882    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189884    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189887    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189889    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189892    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189894    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189897    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189899    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189903    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.191743 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189905    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189908    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189910    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189912    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189915    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189918    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189921    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189923    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189926    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189928    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189931    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189933    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189936    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189939    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189942    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189944    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189947    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189949    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189951    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189954    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.192228 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189956    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189959    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189961    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189963    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189966    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189969    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189971    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189974    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189976    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189978    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189981    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189983    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189986    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189989    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189991    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189994    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.189997    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190000    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190003    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.192702 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190005    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190008    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190010    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190013    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190015    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190019    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190023    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190026    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190029    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190032    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190034    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190037    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190039    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190042    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190044    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190047    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190049    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190052    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190054    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190057    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.193162 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190060    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190454    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190459    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190462    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190464    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190468    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190470    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190473    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190475    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190478    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190481    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190483    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190487    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190491    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190494    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190497    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190500    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190506    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190508    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.193641 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190511    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190513    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190516    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190519    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190521    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190524    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190526    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190529    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190531    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190534    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190536    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190539    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190541    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190545    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190548    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190552    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190555    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190557    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190560    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.194122 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190563    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190566    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190569    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190572    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190574    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190577    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190579    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190582    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190584    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190586    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190589    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190593    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190597    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190599    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190602    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190604    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190607    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190610    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190612    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190614    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.194582 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190617    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190620    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190623    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190626    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190629    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190631    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190634    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190636    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190639    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190642    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190645    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190648    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190651    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190654    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190657    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190659    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190662    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190664    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190667    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190669    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.195089 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190672    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190674    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190676    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190680    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190683    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190685    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190688    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190690    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190693    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190695    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190698    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190700    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190703    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190705    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190708    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190710    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190713    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190715    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190731    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190734    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.195578 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190737    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.190739    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190813    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190821    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190830    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190837    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190844    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190849    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190854    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190858    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190861    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190864    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190868    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190871    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190874    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190878    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190881    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190884    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190886    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190889    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190894    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190897    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190900    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190903    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:04:29.196064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190907    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190910    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190913    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190916    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190920    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190923    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190926    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190929    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190932    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190935    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190939    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190943    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190946    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190948    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190951    2561 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190955    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190959    2561 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190962    2561 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190965    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190968    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190971    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190975    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190978    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190981    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190984    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:04:29.196640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190987    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190990    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190993    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190996    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.190999    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191002    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191005    2561 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191009    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191011    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191019    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191022    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191025    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191028    2561 flags.go:64] FLAG: --help="false"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191031    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191034    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191037    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191040    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191044    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191047    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191050    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191053    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191055    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191058    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:04:29.197329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191065    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191069    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191072    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191074    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191077    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191080    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191083    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191086    2561 flags.go:64] FLAG: --lock-file=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191089    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191092    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191095    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191100    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191103    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191106    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191109    2561 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191112    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191115    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191118    2561 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191122    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191126    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191129    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191133    2561 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191136    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191139    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191142    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:04:29.197943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191145    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191148    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191151    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191154    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191162    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191165    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191168    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191171    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191174    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191180    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191183    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191186    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191190    2561 flags.go:64] FLAG: --port="10250"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191193    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191195    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05917a69f7c019389"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191198    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191201    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191204    2561 flags.go:64] FLAG: --register-node="true"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191207    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191210    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191213    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191216    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191219    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191222    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191225    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:04:29.198556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191229    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191232    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191235    2561 flags.go:64] FLAG: --runonce="false"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191238    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191241    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191244    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191247    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191250    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191253    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191256    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191259    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191261    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191264    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191267    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191269    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191272    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191276    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191280    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191286    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191289    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191291    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191296    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191299    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191301    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191305    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:04:29.199168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191308    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191310    2561 flags.go:64] FLAG: --v="2"
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191314    2561 flags.go:64] FLAG: --version="false"
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191319    2561 flags.go:64] FLAG: --vmodule=""
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191322    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191325    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191441    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191446    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191450    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191453    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191456    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191459    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191462    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191465    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191468    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191470    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191473    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191477    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191480    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191483    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.199852 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191485    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191488    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191490    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191493    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191497    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191500    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191503    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191505    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191508    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191511    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191513    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191516    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191518    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191520    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191523    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191526    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191528    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191531    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191533    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191536    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.200330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191539    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191542    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191545    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191547    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191551    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191554    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191557    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191559    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191562    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191564    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191567    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191569    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191572    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191574    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191577    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191579    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191584    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191591    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191594    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191596    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.200938 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191599    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191602    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191604    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191607    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191609    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191612    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191614    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191617    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191619    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191622    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191625    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191627    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191629    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191632    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191635    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191637    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191639    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191642    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191644    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191647    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.201430 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191649    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191652    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191654    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191657    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191659    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191662    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191664    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191667    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191671    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191674    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191677    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191680    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191683    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191686    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191688    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191691    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191693    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191696    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191698    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191700    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.201920 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191703    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191706    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191708    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191710    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.191713    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.191733    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.199084    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.199102    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.199799    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.199807    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199880    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199886    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199889    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199892    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199895    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.202401 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199898    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199901    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199904    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199906    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199909    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199911    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199914    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199917    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199919    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199922    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199925    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199927    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199930    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199933    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199935    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199938    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199940    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199943    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199945    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199948    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.202788 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199950    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199953    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199955    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199958    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199968    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199971    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199973    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199976    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199979    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199981    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199984    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199987    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199989    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199993    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.199997    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200000    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200003    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200006    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200009    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.203314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200011    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200014    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200016    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200019    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200021    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200024    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200027    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200030    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200032    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200035    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200037    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200040    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200043    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200045    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200048    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200050    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200053    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200056    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200064    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200067    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.203791 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200070    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200072    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200075    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200077    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200080    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200082    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200085    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200087    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200090    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200094    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200097    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200100    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200103    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200106    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200108    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200111    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200114    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200116    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200120    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.204386 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200123    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200126    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200129    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200131    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200134    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200137    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200140    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200143    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200146    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200148    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200151    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200154    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200162    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200165    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200168    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.204993 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200170    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.200175    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200335    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200342    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200345    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200348    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200351    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200355    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200358    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200360    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200363    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200366    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200368    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200371    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200373    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:04:29.205602 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200376    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200378    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200381    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200384    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200386    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200389    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200392    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200395    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200398    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200401    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200405    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200408    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200411    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200414    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200417    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200425    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200428    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200431    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:04:29.206199 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200433    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200436    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200438    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200441    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200443    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200446    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200448    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200451    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200453    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200456    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200458    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200466    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200469    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200471    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200474    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200477    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200480    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200482    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200484    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200487    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:04:29.206753 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200489    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200492    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200495    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200497    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200499    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200502    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200505    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200507    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200509    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200512    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200520    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200523    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200525    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200528    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200530    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200532    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200535    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200537    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200539    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200542    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:04:29.207382 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200544    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200547    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200549    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200551    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200554    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200556    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200559    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200561    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200564    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200566    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200569    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200571    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200573    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200576    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200578    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200581    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200583    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200587    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200591    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200593    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:04:29.207976 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200596    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200598    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200600    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200609    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200611    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200614    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200616    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:29.200619    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.200624    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.200781    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.202759    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.204049    2561 server.go:1019] "Starting client certificate rotation"
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.204165    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:29.208657 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.204224    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:04:29.209005 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.207239    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:29.211906 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.211865    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:04:29.221492 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.221470    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:04:29.224361 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.224347    2561 log.go:25] "Validated CRI v1 image API"
Jul 09 18:04:29.225926 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.225910    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:04:29.228439 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.228418    2561 fs.go:135] Filesystem UUIDs: map[3012a69e-4155-48bf-93b6-904b7d71db56:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b383761d-33bf-4b63-9f4f-c9cfe0400ea9:/dev/nvme0n1p3]
Jul 09 18:04:29.228504 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.228439    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:04:29.233556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.233430    2561 manager.go:216] Machine: {Timestamp:2026-07-09 18:04:29.232215383 +0000 UTC m=+0.333742401 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098106 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2913f20a68d055b3df913fe2c59233 SystemUUID:ec2913f2-0a68-d055-b3df-913fe2c59233 BootID:f4b12322-a57b-440f-a25f-18a55d657e0a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:0a:6f:b4:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:0a:6f:b4:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:0e:ea:99:6d:35 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:04:29.233556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.233540    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:04:29.233703 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.233627    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:04:29.236383 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.236356    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:04:29.236536 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.236386    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-20.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:04:29.236746 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.236712    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:29.237707 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.237695    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:04:29.237778 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.237710    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:04:29.237778 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.237731    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:29.238423 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.238412    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:04:29.239654 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.239644    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:04:29.242367 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242354    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:29.242535 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242526    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:04:29.242581 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242554    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:04:29.242581 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242563    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:04:29.242581 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242578    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:04:29.242684 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242587    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:04:29.242684 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242596    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:04:29.242995 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.242984    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:29.243042 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.243003    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:04:29.245270 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.245246    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:04:29.247184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247164    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:04:29.247251 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247200    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:04:29.247420 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247407    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:04:29.247456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247426    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:04:29.247456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247435    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:04:29.247456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247444    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:04:29.247456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247455    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:04:29.247567 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.247461    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:04:29.248072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248059    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:04:29.248072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248072    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:04:29.248140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248086    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:04:29.248140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248092    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:04:29.248140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248101    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:04:29.248140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.248109    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:04:29.249233 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.249221    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:04:29.249288 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.249234    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:04:29.252653 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.252633    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-20.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:04:29.252748 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.252638    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:04:29.252806 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.252786    2561 server.go:1268] "Started kubelet"
Jul 09 18:04:29.252895 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.252868    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:04:29.256969 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.256755    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:04:29.257161 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.257144    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:04:29.257576 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.257476    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:04:29.257532 ip-10-0-138-20 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:04:29.258634 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.258485    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:04:29.261482 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.261467    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:04:29.263402 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.263075    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zph4r"
Jul 09 18:04:29.263897 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.263875    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:04:29.264327 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.263286    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-20.ec2.internal.18c0b12f52ff5c62  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-20.ec2.internal,UID:ip-10-0-138-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-20.ec2.internal,},FirstTimestamp:2026-07-09 18:04:29.252762722 +0000 UTC m=+0.354289740,LastTimestamp:2026-07-09 18:04:29.252762722 +0000 UTC m=+0.354289740,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-20.ec2.internal,}"
Jul 09 18:04:29.265490 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.265470    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:04:29.265568 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.265474    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:29.265659 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.265642    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.266193 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266166    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:04:29.266193 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266169    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:04:29.266193 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266191    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:04:29.266371 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266241    2561 factory.go:55] Registering systemd factory
Jul 09 18:04:29.266371 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266260    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:04:29.266371 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266290    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:04:29.266371 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266301    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:04:29.266566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266474    2561 factory.go:153] Registering CRI-O factory
Jul 09 18:04:29.266566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266490    2561 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:04:29.266566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266537    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:04:29.266566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266558    2561 factory.go:103] Registering Raw factory
Jul 09 18:04:29.266750 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.266571    2561 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:04:29.267333 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.267316    2561 manager.go:349] Starting recovery of all containers
Jul 09 18:04:29.269333 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.269239    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-20.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:04:29.269755 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.269331    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:04:29.272292 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.272254    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zph4r"
Jul 09 18:04:29.279510 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.279492    2561 manager.go:354] Recovery completed
Jul 09 18:04:29.285378 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.285252    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.287486 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.287468    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.287550 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.287497    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.287550 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.287508    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.288074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288060    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:04:29.288074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288071    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:04:29.288189 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288086    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:04:29.288189 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288088    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.288954 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288941    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.289005 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288971    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.289005 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.288984    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.289262 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.289251    2561 policy_none.go:49] "None policy: Start"
Jul 09 18:04:29.289296 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.289267    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:04:29.289296 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.289277    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:04:29.289878 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.289805    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-20.ec2.internal.18c0b12f55112b12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-20.ec2.internal,UID:ip-10-0-138-20.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-20.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-20.ec2.internal,},FirstTimestamp:2026-07-09 18:04:29.287484178 +0000 UTC m=+0.389011198,LastTimestamp:2026-07-09 18:04:29.287484178 +0000 UTC m=+0.389011198,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-20.ec2.internal,}"
Jul 09 18:04:29.290274 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.290263    2561 policy_none.go:47] "Start"
Jul 09 18:04:29.322370 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322345    2561 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:04:29.322474 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.322386    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:04:29.322474 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322400    2561 server.go:87] "Starting device plugin registration server"
Jul 09 18:04:29.322734 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322705    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:04:29.322811 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322736    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:04:29.322861 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322823    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:04:29.322925 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322917    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:04:29.322978 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.322927    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:04:29.323498 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.323476    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:04:29.323578 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.323533    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.423425 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.423333    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.424679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.424333    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.424679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.424365    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.424679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.424375    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.424679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.424403    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.440256 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.440223    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.440256 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.440254    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-20.ec2.internal\": node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.447448 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.447414    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:04:29.448600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.448583    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:04:29.448695 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.448619    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:04:29.448695 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.448640    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:04:29.448695 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.448670    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:04:29.456885 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.456864    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:29.465115 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.465093    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.549546 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.549493    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"]
Jul 09 18:04:29.549658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.549595    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.550575 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.550560    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.550658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.550589    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.550658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.550600    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.551852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.551841    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.551964 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.551948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.552002 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.551992    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.552567 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552551    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.552655 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552577    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.552655 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552590    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.552655 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552601    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.552655 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552622    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.552655 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.552633    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.553908 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.553893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.553990 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.553925    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:04:29.554678 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.554663    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:04:29.554776 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.554688    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:04:29.554776 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.554701    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:04:29.565686 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.565665    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.567876 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.567861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.567925 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.567883    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.567925 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.567900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ada18c87c87425a91187e5c2f3745abe-config\") pod \"kube-apiserver-proxy-ip-10-0-138-20.ec2.internal\" (UID: \"ada18c87c87425a91187e5c2f3745abe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.581670 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.581646    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-20.ec2.internal\" not found" node="ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.586074 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.586052    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-20.ec2.internal\" not found" node="ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.665811 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.665778    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.668056 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.668111 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.668111 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ada18c87c87425a91187e5c2f3745abe-config\") pod \"kube-apiserver-proxy-ip-10-0-138-20.ec2.internal\" (UID: \"ada18c87c87425a91187e5c2f3745abe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.668174 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.668210 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0b189f46f0aea13312be4240af77808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal\" (UID: \"e0b189f46f0aea13312be4240af77808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.668242 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.668231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ada18c87c87425a91187e5c2f3745abe-config\") pod \"kube-apiserver-proxy-ip-10-0-138-20.ec2.internal\" (UID: \"ada18c87c87425a91187e5c2f3745abe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.766352 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.766285    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.866643 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.866622    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:29.883997 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.883956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.889046 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:29.889026    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:29.966900 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:29.966864    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.067366 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.067300    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.079176 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.079156    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:30.105143 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.105122    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:30.167527 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.167497    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.203802 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.203782    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:04:30.204257 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.203939    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:30.204257 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.203953    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:30.204257 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.203974    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:04:30.268093 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.268067    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.276256 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.276228    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:59:29 +0000 UTC" deadline="2028-01-17 16:02:00.536858163 +0000 UTC"
Jul 09 18:04:30.276256 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.276256    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13365h57m30.260606065s"
Jul 09 18:04:30.368698 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.368665    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.437272 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:30.437233    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b189f46f0aea13312be4240af77808.slice/crio-504600ad1c9f35facf24b9f4c78efa178a8a4f0376e11f669cabb80cb3f8200a WatchSource:0}: Error finding container 504600ad1c9f35facf24b9f4c78efa178a8a4f0376e11f669cabb80cb3f8200a: Status 404 returned error can't find the container with id 504600ad1c9f35facf24b9f4c78efa178a8a4f0376e11f669cabb80cb3f8200a
Jul 09 18:04:30.441985 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.441970    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:04:30.452106 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.452067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal" event={"ID":"e0b189f46f0aea13312be4240af77808","Type":"ContainerStarted","Data":"504600ad1c9f35facf24b9f4c78efa178a8a4f0376e11f669cabb80cb3f8200a"}
Jul 09 18:04:30.469375 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.469353    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.569849 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:30.569817    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-20.ec2.internal\" not found"
Jul 09 18:04:30.665496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.665428    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:30.666347 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.666328    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:30.675220 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:30.675196    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada18c87c87425a91187e5c2f3745abe.slice/crio-3e4c3f1b7cc27de756c17f9671a9e1a2cdeb4bb45ec48170cfc16c82905340e4 WatchSource:0}: Error finding container 3e4c3f1b7cc27de756c17f9671a9e1a2cdeb4bb45ec48170cfc16c82905340e4: Status 404 returned error can't find the container with id 3e4c3f1b7cc27de756c17f9671a9e1a2cdeb4bb45ec48170cfc16c82905340e4
Jul 09 18:04:30.686743 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.686714    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:30.687779 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.687765    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal"
Jul 09 18:04:30.696293 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:30.696275    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:04:31.243782 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.243640    2561 apiserver.go:52] "Watching apiserver"
Jul 09 18:04:31.254236 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.254206    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:04:31.254963 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.254803    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-zjs8g","openshift-dns/node-resolver-vk8gc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal","openshift-multus/multus-4p5sn","openshift-network-diagnostics/network-check-target-2dwcb","openshift-ovn-kubernetes/ovnkube-node-5t7cr","kube-system/global-pull-secret-syncer-zfwnv","kube-system/konnectivity-agent-xrz68","kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d","openshift-image-registry/node-ca-szr88","openshift-multus/multus-additional-cni-plugins-k7vpg","openshift-multus/network-metrics-daemon-blwkj","openshift-network-operator/iptables-alerter-8cgn4"]
Jul 09 18:04:31.256466 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.256436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.256578 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.256516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:31.257606 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.257583    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.257715 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.257660    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:31.258861 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.258844    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.260453 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.260261    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.261106 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.261085    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwjl\""
Jul 09 18:04:31.261291 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.261222    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.261389 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.261342    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:04:31.261449 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.261433    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.261545 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.261519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:31.261652 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.261587    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:31.262503 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.262484    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.263152 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.262836    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-flpst\""
Jul 09 18:04:31.263152 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.262972    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.263152 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.263094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:04:31.263353 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.263179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.263353 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.263280    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:04:31.264597 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.264544    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.265367 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.265341    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.265634 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.265565    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.265906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.265915    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.266169    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.266303    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.266487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.266689    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:04:31.266870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.266872    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7trbd\""
Jul 09 18:04:31.269468 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.268955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.270182 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.269766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h6pfk\""
Jul 09 18:04:31.270182 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.269773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:04:31.270182 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.269843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.270182 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.269895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:04:31.270182 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.270019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dndp\""
Jul 09 18:04:31.270469 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.270208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.270469 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.270463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.271040 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.271019    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.271196 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.271177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-58jcx\""
Jul 09 18:04:31.272432 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.271605    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.272432 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.271780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.272961 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.272942    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:04:31.273096 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.273082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.273189 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.273175    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.273452 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.273435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.274216 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274198    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:04:31.274352 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274332    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j49ws\""
Jul 09 18:04:31.274422 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274404    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:04:31.274478 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z762x\" (UniqueName: \"kubernetes.io/projected/79d1e3ce-e800-4992-a055-50acfe377bf3-kube-api-access-z762x\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.274531 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-cni-binary-copy\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274531 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-systemd\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.274531 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-var-lib-kubelet\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.274675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-system-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274572    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.274675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-host\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.274675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdz4\" (UniqueName: \"kubernetes.io/projected/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-kube-api-access-6pdz4\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysconfig\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-systemd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-os-release\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-socket-dir-parent\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7x97p\""
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-bin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-daemon-config\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.274894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-ovn\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-script-lib\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.274960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-netd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-conf-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-conf\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-lib-modules\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-tmp\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79d1e3ce-e800-4992-a055-50acfe377bf3-host-slash\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-cnibin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-env-overrides\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb87d44a-492c-434b-8dd9-23798314b545-ovn-node-metrics-cert\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qts\" (UniqueName: \"kubernetes.io/projected/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-kube-api-access-74qts\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-multus-certs\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-netns\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkd7\" (UniqueName: \"kubernetes.io/projected/fb87d44a-492c-434b-8dd9-23798314b545-kube-api-access-ldkd7\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzm2\" (UniqueName: \"kubernetes.io/projected/c5f59614-c402-4162-87a3-ab09262ca067-kube-api-access-4wzm2\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-var-lib-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-kubelet-config\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275529    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-agent-certs\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79d1e3ce-e800-4992-a055-50acfe377bf3-iptables-alerter-script\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-hostroot\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-slash\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-node-log\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-hosts-file\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-kubernetes\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-k8s-cni-cncf-io\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-tuned\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-netns\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.275918 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-multus\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275901    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-etc-kubernetes\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275927    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-dbus\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.275995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-konnectivity-ca\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-kubelet\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-tmp-dir\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276130    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-kubelet\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7j9p\" (UniqueName: \"kubernetes.io/projected/181c05ea-94cb-43d5-8250-0b16e9765f70-kube-api-access-c7j9p\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-systemd-units\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-etc-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-log-socket\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-bin\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-config\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.276683 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.277434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-modprobe-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.277434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-run\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.277434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.276412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-sys\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.279401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.278296    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fl585\""
Jul 09 18:04:31.279401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.278592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:04:31.280640 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.280486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:04:31.281782 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.280767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:04:31.285120 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.285097    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:04:31.306679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.306656    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sb4dv"
Jul 09 18:04:31.315612 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.315563    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sb4dv"
Jul 09 18:04:31.367986 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.367959    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:04:31.377130 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-var-lib-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-kubelet-config\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377162    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377188    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94sn\" (UniqueName: \"kubernetes.io/projected/87db40f4-a668-401e-9eb0-737aba2f6a03-kube-api-access-x94sn\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-var-lib-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-agent-certs\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.377287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377237    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-kubelet-config\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79d1e3ce-e800-4992-a055-50acfe377bf3-iptables-alerter-script\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-hostroot\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-slash\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-node-log\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-hosts-file\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-kubernetes\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-k8s-cni-cncf-io\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-slash\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-hostroot\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.377605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-k8s-cni-cncf-io\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377616    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-node-log\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-kubernetes\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377640    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-hosts-file\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-tuned\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87db40f4-a668-401e-9eb0-737aba2f6a03-serviceca\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cnibin\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-netns\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-multus\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79d1e3ce-e800-4992-a055-50acfe377bf3-iptables-alerter-script\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-etc-kubernetes\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.377986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-dbus\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-konnectivity-ca\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-multus\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-etc-kubernetes\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-netns\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c451f403-441f-4d69-992a-e55962af5083-dbus\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-kubelet\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-tmp-dir\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-registration-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-kubelet\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7j9p\" (UniqueName: \"kubernetes.io/projected/181c05ea-94cb-43d5-8250-0b16e9765f70-kube-api-access-c7j9p\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-systemd-units\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-etc-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-kubelet\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-log-socket\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.378970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-bin\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-kubelet\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-config\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkg4x\" (UniqueName: \"kubernetes.io/projected/427595fe-d16b-4d25-a928-be65fd76c7ad-kube-api-access-fkg4x\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-modprobe-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-run\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-sys\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-konnectivity-ca\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87db40f4-a668-401e-9eb0-737aba2f6a03-host\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-tmp-dir\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-bin\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z762x\" (UniqueName: \"kubernetes.io/projected/79d1e3ce-e800-4992-a055-50acfe377bf3-kube-api-access-z762x\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-cni-binary-copy\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.378876    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.378886    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-systemd\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.379771 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.378947    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:31.878927405 +0000 UTC m=+2.980454424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-systemd\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.378966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-var-lib-kubelet\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-var-lib-kubelet\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-modprobe-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.379037    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:31.879026398 +0000 UTC m=+2.980553409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-system-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-systemd-units\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379092    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-log-socket\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-host\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379149    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-host\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdz4\" (UniqueName: \"kubernetes.io/projected/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-kube-api-access-6pdz4\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysconfig\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-systemd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-socket-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.380610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-etc-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-device-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-os-release\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-run\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-socket-dir-parent\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-systemd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysconfig\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-sys\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-d\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-bin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-config\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-daemon-config\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-socket-dir-parent\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-ovn\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-system-cni-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-script-lib\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.381427 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-os-release\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-openvswitch\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-var-lib-cni-bin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4v4\" (UniqueName: \"kubernetes.io/projected/5149ade1-96ea-4336-90a2-3b0f53c2e8da-kube-api-access-lk4v4\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-netd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-conf-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.379984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-conf\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-cni-binary-copy\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-lib-modules\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-tmp\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-os-release\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-cni-netd\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79d1e3ce-e800-4992-a055-50acfe377bf3-host-slash\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-daemon-config\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-multus-conf-dir\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-run-ovn\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-cnibin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.382297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-env-overrides\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb87d44a-492c-434b-8dd9-23798314b545-ovn-node-metrics-cert\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-sysctl-conf\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-ovnkube-script-lib\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qts\" (UniqueName: \"kubernetes.io/projected/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-kube-api-access-74qts\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-sys-fs\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-lib-modules\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-multus-certs\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-netns\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkd7\" (UniqueName: \"kubernetes.io/projected/fb87d44a-492c-434b-8dd9-23798314b545-kube-api-access-ldkd7\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzm2\" (UniqueName: \"kubernetes.io/projected/c5f59614-c402-4162-87a3-ab09262ca067-kube-api-access-4wzm2\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb87d44a-492c-434b-8dd9-23798314b545-env-overrides\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79d1e3ce-e800-4992-a055-50acfe377bf3-host-slash\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-cnibin\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/181c05ea-94cb-43d5-8250-0b16e9765f70-host-run-multus-certs\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.380967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb87d44a-492c-434b-8dd9-23798314b545-host-run-netns\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.382495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10-agent-certs\") pod \"konnectivity-agent-xrz68\" (UID: \"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10\") " pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.383995 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.383333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-etc-tuned\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.383995 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.383594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb87d44a-492c-434b-8dd9-23798314b545-ovn-node-metrics-cert\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.383995 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.383811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-tmp\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.390248 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.390226    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:31.390376 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.390252    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:31.390376 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.390266    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:31.390376 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.390319    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:31.890301244 +0000 UTC m=+2.991828271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:31.393063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.392998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7j9p\" (UniqueName: \"kubernetes.io/projected/181c05ea-94cb-43d5-8250-0b16e9765f70-kube-api-access-c7j9p\") pod \"multus-4p5sn\" (UID: \"181c05ea-94cb-43d5-8250-0b16e9765f70\") " pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.394292 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.394222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdz4\" (UniqueName: \"kubernetes.io/projected/c2afda61-2c05-44c1-a4ff-c73ef0b46e2d-kube-api-access-6pdz4\") pod \"node-resolver-vk8gc\" (UID: \"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d\") " pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.396553 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.396181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkd7\" (UniqueName: \"kubernetes.io/projected/fb87d44a-492c-434b-8dd9-23798314b545-kube-api-access-ldkd7\") pod \"ovnkube-node-5t7cr\" (UID: \"fb87d44a-492c-434b-8dd9-23798314b545\") " pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.396553 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.396488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z762x\" (UniqueName: \"kubernetes.io/projected/79d1e3ce-e800-4992-a055-50acfe377bf3-kube-api-access-z762x\") pod \"iptables-alerter-8cgn4\" (UID: \"79d1e3ce-e800-4992-a055-50acfe377bf3\") " pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.396712 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.396595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qts\" (UniqueName: \"kubernetes.io/projected/5e973edd-61b9-4a2d-b95d-31f6bdb249ab-kube-api-access-74qts\") pod \"tuned-zjs8g\" (UID: \"5e973edd-61b9-4a2d-b95d-31f6bdb249ab\") " pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.397643 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.397594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzm2\" (UniqueName: \"kubernetes.io/projected/c5f59614-c402-4162-87a3-ab09262ca067-kube-api-access-4wzm2\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.454048 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.454005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal" event={"ID":"ada18c87c87425a91187e5c2f3745abe","Type":"ContainerStarted","Data":"3e4c3f1b7cc27de756c17f9671a9e1a2cdeb4bb45ec48170cfc16c82905340e4"}
Jul 09 18:04:31.481136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4v4\" (UniqueName: \"kubernetes.io/projected/5149ade1-96ea-4336-90a2-3b0f53c2e8da-kube-api-access-lk4v4\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-os-release\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-sys-fs\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94sn\" (UniqueName: \"kubernetes.io/projected/87db40f4-a668-401e-9eb0-737aba2f6a03-kube-api-access-x94sn\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87db40f4-a668-401e-9eb0-737aba2f6a03-serviceca\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cnibin\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-sys-fs\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-os-release\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cnibin\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-registration-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkg4x\" (UniqueName: \"kubernetes.io/projected/427595fe-d16b-4d25-a928-be65fd76c7ad-kube-api-access-fkg4x\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87db40f4-a668-401e-9eb0-737aba2f6a03-host\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-socket-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-device-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.481900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-device-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-registration-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87db40f4-a668-401e-9eb0-737aba2f6a03-serviceca\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481944    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87db40f4-a668-401e-9eb0-737aba2f6a03-host\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.481962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5149ade1-96ea-4336-90a2-3b0f53c2e8da-system-cni-dir\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.482047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/427595fe-d16b-4d25-a928-be65fd76c7ad-socket-dir\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.482070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.482070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.482481 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.482093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5149ade1-96ea-4336-90a2-3b0f53c2e8da-cni-binary-copy\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.491762 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.491717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4v4\" (UniqueName: \"kubernetes.io/projected/5149ade1-96ea-4336-90a2-3b0f53c2e8da-kube-api-access-lk4v4\") pod \"multus-additional-cni-plugins-k7vpg\" (UID: \"5149ade1-96ea-4336-90a2-3b0f53c2e8da\") " pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.492203 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.492183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkg4x\" (UniqueName: \"kubernetes.io/projected/427595fe-d16b-4d25-a928-be65fd76c7ad-kube-api-access-fkg4x\") pod \"aws-ebs-csi-driver-node-ggx9d\" (UID: \"427595fe-d16b-4d25-a928-be65fd76c7ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.492276 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.492184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94sn\" (UniqueName: \"kubernetes.io/projected/87db40f4-a668-401e-9eb0-737aba2f6a03-kube-api-access-x94sn\") pod \"node-ca-szr88\" (UID: \"87db40f4-a668-401e-9eb0-737aba2f6a03\") " pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.574687 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.574598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8cgn4"
Jul 09 18:04:31.586654 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.586616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p5sn"
Jul 09 18:04:31.595412 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.595390    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:31.602091 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.602073    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:31.608799 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.608779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zjs8g"
Jul 09 18:04:31.616437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.616418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vk8gc"
Jul 09 18:04:31.624233 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.624211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d"
Jul 09 18:04:31.630803 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.630782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szr88"
Jul 09 18:04:31.642455 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.642423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k7vpg"
Jul 09 18:04:31.785359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.785324    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:31.885616 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.885521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:31.885616 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.885570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:31.885874 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.885685    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:31.885874 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.885691    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:31.885874 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.885781    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.885763175 +0000 UTC m=+3.987290200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:31.885874 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.885853    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.885835673 +0000 UTC m=+3.987362690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:31.986539 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:31.986500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:31.986751 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.986676    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:31.986751 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.986701    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:31.986751 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.986714    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:31.986914 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:31.986799    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:32.986777508 +0000 UTC m=+4.088304533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:32.068861 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.068831    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:32.317998 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.317904    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:31 +0000 UTC" deadline="2028-01-19 06:32:05.701905177 +0000 UTC"
Jul 09 18:04:32.317998 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.317949    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13404h27m33.383960186s"
Jul 09 18:04:32.422358 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.422328    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d1e3ce_e800_4992_a055_50acfe377bf3.slice/crio-15e63557663355ac9100f1c11824e8ed7f72704958b092866663416f8ae8a221 WatchSource:0}: Error finding container 15e63557663355ac9100f1c11824e8ed7f72704958b092866663416f8ae8a221: Status 404 returned error can't find the container with id 15e63557663355ac9100f1c11824e8ed7f72704958b092866663416f8ae8a221
Jul 09 18:04:32.434161 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.434130    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5149ade1_96ea_4336_90a2_3b0f53c2e8da.slice/crio-795f7af615aac5810a0baf2dc8e94a443b7e6dada56a92b7f4ecd881b1a5d78b WatchSource:0}: Error finding container 795f7af615aac5810a0baf2dc8e94a443b7e6dada56a92b7f4ecd881b1a5d78b: Status 404 returned error can't find the container with id 795f7af615aac5810a0baf2dc8e94a443b7e6dada56a92b7f4ecd881b1a5d78b
Jul 09 18:04:32.434785 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.434766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdc5c26_f7d4_4ae7_aead_a7c7a7934c10.slice/crio-fb94c26752c8268878d2f18726d5e385a12d4aef87611c5d0a29ba67d9099b6e WatchSource:0}: Error finding container fb94c26752c8268878d2f18726d5e385a12d4aef87611c5d0a29ba67d9099b6e: Status 404 returned error can't find the container with id fb94c26752c8268878d2f18726d5e385a12d4aef87611c5d0a29ba67d9099b6e
Jul 09 18:04:32.439314 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.439294    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e973edd_61b9_4a2d_b95d_31f6bdb249ab.slice/crio-1e1fe09dcfb85e0003e9303a3c81e31286209c3cd8e9e9ef8a6b4ddd6e42759c WatchSource:0}: Error finding container 1e1fe09dcfb85e0003e9303a3c81e31286209c3cd8e9e9ef8a6b4ddd6e42759c: Status 404 returned error can't find the container with id 1e1fe09dcfb85e0003e9303a3c81e31286209c3cd8e9e9ef8a6b4ddd6e42759c
Jul 09 18:04:32.440568 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.440548    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87db40f4_a668_401e_9eb0_737aba2f6a03.slice/crio-7183a9101aba0199f06e90a62c07f45d6c1ca4256f9ee7a3eca387f0ccaf09d5 WatchSource:0}: Error finding container 7183a9101aba0199f06e90a62c07f45d6c1ca4256f9ee7a3eca387f0ccaf09d5: Status 404 returned error can't find the container with id 7183a9101aba0199f06e90a62c07f45d6c1ca4256f9ee7a3eca387f0ccaf09d5
Jul 09 18:04:32.442107 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.442086    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427595fe_d16b_4d25_a928_be65fd76c7ad.slice/crio-588bbbbd48ab49494dedb2f0f18b1a21dba4cf44f7f39d8e14e374c3d43c2b8d WatchSource:0}: Error finding container 588bbbbd48ab49494dedb2f0f18b1a21dba4cf44f7f39d8e14e374c3d43c2b8d: Status 404 returned error can't find the container with id 588bbbbd48ab49494dedb2f0f18b1a21dba4cf44f7f39d8e14e374c3d43c2b8d
Jul 09 18:04:32.442760 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.442738    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2afda61_2c05_44c1_a4ff_c73ef0b46e2d.slice/crio-e04008d40eceff390f6ecb82d457b4a1c0313e16133062842a5c23e3c82386e3 WatchSource:0}: Error finding container e04008d40eceff390f6ecb82d457b4a1c0313e16133062842a5c23e3c82386e3: Status 404 returned error can't find the container with id e04008d40eceff390f6ecb82d457b4a1c0313e16133062842a5c23e3c82386e3
Jul 09 18:04:32.444330 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.444297    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181c05ea_94cb_43d5_8250_0b16e9765f70.slice/crio-a115e7970fa0c1c86e364482794264d685fd9f074488a300d94a44e61de8cc3d WatchSource:0}: Error finding container a115e7970fa0c1c86e364482794264d685fd9f074488a300d94a44e61de8cc3d: Status 404 returned error can't find the container with id a115e7970fa0c1c86e364482794264d685fd9f074488a300d94a44e61de8cc3d
Jul 09 18:04:32.445584 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:04:32.445387    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb87d44a_492c_434b_8dd9_23798314b545.slice/crio-438333b6b6e60e012dd83b1ba27ac4ebd93c757afdd2ff5a8ea8847331e5a433 WatchSource:0}: Error finding container 438333b6b6e60e012dd83b1ba27ac4ebd93c757afdd2ff5a8ea8847331e5a433: Status 404 returned error can't find the container with id 438333b6b6e60e012dd83b1ba27ac4ebd93c757afdd2ff5a8ea8847331e5a433
Jul 09 18:04:32.449387 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.449325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:32.449445 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.449325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:32.449483 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.449461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:32.449569 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.449501    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:32.456669 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.456647    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8cgn4" event={"ID":"79d1e3ce-e800-4992-a055-50acfe377bf3","Type":"ContainerStarted","Data":"15e63557663355ac9100f1c11824e8ed7f72704958b092866663416f8ae8a221"}
Jul 09 18:04:32.458633 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.458607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"438333b6b6e60e012dd83b1ba27ac4ebd93c757afdd2ff5a8ea8847331e5a433"}
Jul 09 18:04:32.460972 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.460946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p5sn" event={"ID":"181c05ea-94cb-43d5-8250-0b16e9765f70","Type":"ContainerStarted","Data":"a115e7970fa0c1c86e364482794264d685fd9f074488a300d94a44e61de8cc3d"}
Jul 09 18:04:32.462117 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.462084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vk8gc" event={"ID":"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d","Type":"ContainerStarted","Data":"e04008d40eceff390f6ecb82d457b4a1c0313e16133062842a5c23e3c82386e3"}
Jul 09 18:04:32.463059 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.463036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjs8g" event={"ID":"5e973edd-61b9-4a2d-b95d-31f6bdb249ab","Type":"ContainerStarted","Data":"1e1fe09dcfb85e0003e9303a3c81e31286209c3cd8e9e9ef8a6b4ddd6e42759c"}
Jul 09 18:04:32.464342 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.464318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerStarted","Data":"795f7af615aac5810a0baf2dc8e94a443b7e6dada56a92b7f4ecd881b1a5d78b"}
Jul 09 18:04:32.465410 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.465386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xrz68" event={"ID":"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10","Type":"ContainerStarted","Data":"fb94c26752c8268878d2f18726d5e385a12d4aef87611c5d0a29ba67d9099b6e"}
Jul 09 18:04:32.466415 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.466393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d" event={"ID":"427595fe-d16b-4d25-a928-be65fd76c7ad","Type":"ContainerStarted","Data":"588bbbbd48ab49494dedb2f0f18b1a21dba4cf44f7f39d8e14e374c3d43c2b8d"}
Jul 09 18:04:32.467501 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.467476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szr88" event={"ID":"87db40f4-a668-401e-9eb0-737aba2f6a03","Type":"ContainerStarted","Data":"7183a9101aba0199f06e90a62c07f45d6c1ca4256f9ee7a3eca387f0ccaf09d5"}
Jul 09 18:04:32.893074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.892779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:32.893074 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.893024    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.893074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.893050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:32.893368 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.893105    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:34.893083837 +0000 UTC m=+5.994610864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:32.893368 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.893167    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.893368 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:32.893216    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:34.893202273 +0000 UTC m=+5.994729293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:32.994339 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:32.994283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:33.002042 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:33.002004    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:33.002042 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:33.002047    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:33.002283 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:33.002062    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:33.002283 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:33.002140    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:35.002116894 +0000 UTC m=+6.103643923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:33.181528 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.181142    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:04:33.318774 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.318699    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:59:31 +0000 UTC" deadline="2028-01-23 21:42:56.674120403 +0000 UTC"
Jul 09 18:04:33.318774 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.318772    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13515h38m23.355352198s"
Jul 09 18:04:33.451938 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.449108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:33.451938 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:33.449238    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:33.487356 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.487314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal" event={"ID":"ada18c87c87425a91187e5c2f3745abe","Type":"ContainerStarted","Data":"62b3c98321b4fa6f17bff136400355a374dabaca6cc4bc38e812f529eb40cddf"}
Jul 09 18:04:33.508529 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.508492    2561 generic.go:356] "Generic (PLEG): container finished" podID="e0b189f46f0aea13312be4240af77808" containerID="4a51e02d9afab6f496a3366bd161f9a0ac3a127e1803f42ee1c13538f7fea7f5" exitCode=0
Jul 09 18:04:33.508709 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.508540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal" event={"ID":"e0b189f46f0aea13312be4240af77808","Type":"ContainerDied","Data":"4a51e02d9afab6f496a3366bd161f9a0ac3a127e1803f42ee1c13538f7fea7f5"}
Jul 09 18:04:33.526924 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:33.526647    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-20.ec2.internal" podStartSLOduration=3.526627952 podStartE2EDuration="3.526627952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:33.504190173 +0000 UTC m=+4.605717201" watchObservedRunningTime="2026-07-09 18:04:33.526627952 +0000 UTC m=+4.628154981"
Jul 09 18:04:34.449188 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:34.449145    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:34.449648 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.449295    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:34.449764 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:34.449690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:34.449838 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.449793    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:34.532456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:34.532019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal" event={"ID":"e0b189f46f0aea13312be4240af77808","Type":"ContainerStarted","Data":"7b5a69f9ff091db8b2e39be05b7623bd2fd2b2a778b3f887887df3cb1ea27a91"}
Jul 09 18:04:34.909139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:34.909041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:34.909139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:34.909097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:34.909370 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.909222    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:34.909370 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.909287    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:38.909266519 +0000 UTC m=+10.010793757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:34.909778 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.909715    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:34.909863 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:34.909792    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:38.909777188 +0000 UTC m=+10.011304208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:35.010530 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:35.010105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:35.010530 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:35.010316    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:35.010530 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:35.010349    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:35.010530 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:35.010364    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:35.010530 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:35.010423    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:39.010404333 +0000 UTC m=+10.111931341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:35.462750 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:35.462673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:35.463774 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:35.463379    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:36.449044 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:36.449001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:36.449223 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:36.449191    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:36.449635 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:36.449579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:36.449994 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:36.449954    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:37.449136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:37.449057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:37.449606 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:37.449196    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:38.449026 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:38.448991    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:38.449185 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.449118    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:38.449185 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:38.449167    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:38.449625 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.449304    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:38.947227 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:38.947107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:38.947227 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:38.947169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:38.947441 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.947235    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:38.947441 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.947255    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:38.947441 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.947303    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:46.947286692 +0000 UTC m=+18.048813711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:38.947441 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:38.947322    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:04:46.947313158 +0000 UTC m=+18.048840177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:39.048507 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:39.048314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:39.048681 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:39.048515    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:39.048681 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:39.048539    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:39.048681 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:39.048553    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.048681 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:39.048622    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:04:47.048602617 +0000 UTC m=+18.150129641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:39.450669 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:39.450634    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:39.451166 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:39.450769    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:40.449898 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:40.449857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:40.450092 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:40.449998    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:40.450348 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:40.450329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:40.450514 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:40.450473    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:41.449621 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:41.449585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:41.450035 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:41.449853    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:42.449196 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:42.449158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:42.449409 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:42.449164    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:42.449409 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:42.449298    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:42.449529 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:42.449419    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:43.449376 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:43.449335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:43.449838 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:43.449461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:44.449395 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:44.449359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:44.449832 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:44.449359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:44.449832 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:44.449507    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:44.449832 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:44.449580    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:45.449244 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:45.449204    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:45.449430 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:45.449364    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:46.449901 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:46.449862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:46.450364 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:46.449974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:46.450364 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:46.450029    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:46.450364 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:46.450138    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:47.012879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:47.012829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:47.013055 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:47.012930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:47.013055 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.012998    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:47.013055 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.013036    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:47.013270 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.013079    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:03.013056022 +0000 UTC m=+34.114583044 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:04:47.013270 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.013100    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:03.013089796 +0000 UTC m=+34.114616806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:04:47.113574 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:47.113521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:47.113792 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.113703    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:04:47.113792 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.113750    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:04:47.113792 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.113764    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:47.113951 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.113830    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:03.113810317 +0000 UTC m=+34.215337340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:04:47.449644 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:47.449553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:47.449829 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:47.449665    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:48.449952 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:48.449907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:48.450390 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:48.449963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:48.450390 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:48.450071    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:48.450390 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:48.450211    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:49.450417 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:49.450386    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:49.450902 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:49.450506    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:50.448905 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:50.448852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:50.449085 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:50.448991    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:50.449085 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:50.449037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:50.449190 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:50.449155    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:51.449361 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.449194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:51.449886 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:51.449466    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:51.571058 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.571024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d" event={"ID":"427595fe-d16b-4d25-a928-be65fd76c7ad","Type":"ContainerStarted","Data":"3deb9a82e09d7344b9b4f967c9c853b24f9c32cfb9d00023a4d698620cb161da"}
Jul 09 18:04:51.572673 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.572632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szr88" event={"ID":"87db40f4-a668-401e-9eb0-737aba2f6a03","Type":"ContainerStarted","Data":"e3dea35ab65c073724aa654aa08f6872daf1b244bfef76774bb871feaf505c7d"}
Jul 09 18:04:51.575717 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.575685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"2db8705c7f946cb2c723c736e342cc0f8a8621bc1d57d28967c260c7cf53c8af"}
Jul 09 18:04:51.578155 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.578129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p5sn" event={"ID":"181c05ea-94cb-43d5-8250-0b16e9765f70","Type":"ContainerStarted","Data":"bea242006d8923606e9d3eb968df8ad98d54ad11c7cb68fb26c123f16223940e"}
Jul 09 18:04:51.579766 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.579738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vk8gc" event={"ID":"c2afda61-2c05-44c1-a4ff-c73ef0b46e2d","Type":"ContainerStarted","Data":"f4c862104b1ab378296a50170dd103d67a4ecc190a3a2c8de2306023fcc91eea"}
Jul 09 18:04:51.581299 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.581272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zjs8g" event={"ID":"5e973edd-61b9-4a2d-b95d-31f6bdb249ab","Type":"ContainerStarted","Data":"9f1093690befa6212eb2594cd518335c10c9db2f40e0896e0b6d4dacc7409284"}
Jul 09 18:04:51.582985 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.582962    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="ab30aedb4d4200e9e823948c392e2d6e80203c42127e22c6115cb22a27cfe8d8" exitCode=0
Jul 09 18:04:51.583110 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.583027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"ab30aedb4d4200e9e823948c392e2d6e80203c42127e22c6115cb22a27cfe8d8"}
Jul 09 18:04:51.584580 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.584555    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xrz68" event={"ID":"6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10","Type":"ContainerStarted","Data":"5cb99f7c5cf503c14c2a547f4a3981e69d4be1ad982a2c97ebb20755712cce3f"}
Jul 09 18:04:51.589509 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.589467    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-szr88" podStartSLOduration=3.780019478 podStartE2EDuration="22.589456658s" totalImagesPullingTime="18.80943718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:51.589382568 +0000 UTC m=+22.690909595" watchObservedRunningTime="2026-07-09 18:04:51.589456658 +0000 UTC m=+22.690983685"
Jul 09 18:04:51.589770 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.589747    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-20.ec2.internal" podStartSLOduration=21.589743143 podStartE2EDuration="21.589743143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:34.552137367 +0000 UTC m=+5.653664394" watchObservedRunningTime="2026-07-09 18:04:51.589743143 +0000 UTC m=+22.691270170"
Jul 09 18:04:51.603792 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.603755    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xrz68" podStartSLOduration=11.927437996 podStartE2EDuration="22.603738849s" totalImagesPullingTime="10.676300853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:51.603698601 +0000 UTC m=+22.705225639" watchObservedRunningTime="2026-07-09 18:04:51.603738849 +0000 UTC m=+22.705265871"
Jul 09 18:04:51.623911 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.623870    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zjs8g" podStartSLOduration=3.8103108629999998 podStartE2EDuration="22.6238543s" totalImagesPullingTime="18.813543437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:51.623550116 +0000 UTC m=+22.725077139" watchObservedRunningTime="2026-07-09 18:04:51.6238543 +0000 UTC m=+22.725381318"
Jul 09 18:04:51.640457 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.640414    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vk8gc" podStartSLOduration=3.834364612 podStartE2EDuration="22.640398775s" totalImagesPullingTime="18.806034163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:51.639806074 +0000 UTC m=+22.741333103" watchObservedRunningTime="2026-07-09 18:04:51.640398775 +0000 UTC m=+22.741925801"
Jul 09 18:04:51.657444 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:51.657399    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4p5sn" podStartSLOduration=3.784698161 podStartE2EDuration="22.657384558s" totalImagesPullingTime="18.872686397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:51.657016566 +0000 UTC m=+22.758543592" watchObservedRunningTime="2026-07-09 18:04:51.657384558 +0000 UTC m=+22.758911585"
Jul 09 18:04:52.449136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.448888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:52.449269 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.448888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:52.449325 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:52.449267    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:52.449387 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:52.449341    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:52.589576 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.589477    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8cgn4" event={"ID":"79d1e3ce-e800-4992-a055-50acfe377bf3","Type":"ContainerStarted","Data":"3fa4ae92286dc028af7aa0f0638901f1ecb64bbfe71075039464e3491c1b7e51"}
Jul 09 18:04:52.593096 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:04:52.593578 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593546    2561 generic.go:356] "Generic (PLEG): container finished" podID="fb87d44a-492c-434b-8dd9-23798314b545" containerID="e453fdfce9b244d0c9b727f5dc7b11f6dd1d37cdf3c1400e6ee5a79592ef81ee" exitCode=1
Jul 09 18:04:52.593786 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"dd6a8c48710fdbf2bfa4b4aa4762f78022ec9e0cd18d919f9da8ddbc8fefa622"}
Jul 09 18:04:52.593786 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593675    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"b9cae90c0d6bfa98e00c8303caa4f513c393b71a059e1eb56289186f86816939"}
Jul 09 18:04:52.593786 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"f5308f671c611318f0726487995f38983415443ef8f54c483618466246b01950"}
Jul 09 18:04:52.593984 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"a8761f6bd9dcb59a03f43bd00dc5b1cb8bfdec6256a10bc75c1b3ee034c91890"}
Jul 09 18:04:52.593984 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.593845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerDied","Data":"e453fdfce9b244d0c9b727f5dc7b11f6dd1d37cdf3c1400e6ee5a79592ef81ee"}
Jul 09 18:04:52.606069 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:52.606013    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8cgn4" podStartSLOduration=4.785680539 podStartE2EDuration="23.605991321s" totalImagesPullingTime="18.820310782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:52.605054906 +0000 UTC m=+23.706581947" watchObservedRunningTime="2026-07-09 18:04:52.605991321 +0000 UTC m=+23.707518349"
Jul 09 18:04:53.449401 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:53.449318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:53.449906 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:53.449455    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:53.460327 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:53.460284    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:04:53.598088 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:53.598000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d" event={"ID":"427595fe-d16b-4d25-a928-be65fd76c7ad","Type":"ContainerStarted","Data":"53e4cea0e9b33260e247c98f5c679c78b6f55967b6c18ff2b541d83160419ffa"}
Jul 09 18:04:53.993155 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:53.993074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:53.993802 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:53.993783    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:04:54.337294 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.337162    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:04:53.460307229Z","UUID":"18dc02e6-dcc0-4f95-8807-9d82207f6d5b","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:04:54.338954 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.338930    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:04:54.339078 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.338977    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:04:54.449418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.449381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:54.449418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.449401    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:54.449974 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:54.449509    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:54.449974 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:54.449659    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:54.603463 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.603394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:04:54.603877 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:54.603851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"df80e65a0fbc502d9b8161977af46625b1fb608291c777790b4e76e008884744"}
Jul 09 18:04:55.449277 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:55.449087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:55.449465 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:55.449375    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:56.449134 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.449108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:56.449284 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.449107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:56.449284 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:56.449270    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:56.449423 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:56.449399    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:56.611816 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.611625    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:04:56.612578 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.612148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"18babcbe94fba0c3cc1a76b0ca30aa21d7a0518410476ff1d00c782a95eebbfb"}
Jul 09 18:04:56.612578 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.612400    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:56.612578 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.612559    2561 scope.go:117] "RemoveContainer" containerID="e453fdfce9b244d0c9b727f5dc7b11f6dd1d37cdf3c1400e6ee5a79592ef81ee"
Jul 09 18:04:56.614102 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.614080    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="32929462fdc564007bef0b1ef023c1a4112d15d2c071d47be4433b38e3eba73f" exitCode=0
Jul 09 18:04:56.614229 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.614146    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"32929462fdc564007bef0b1ef023c1a4112d15d2c071d47be4433b38e3eba73f"}
Jul 09 18:04:56.616344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.616322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d" event={"ID":"427595fe-d16b-4d25-a928-be65fd76c7ad","Type":"ContainerStarted","Data":"91cb8daba620eaf6c678e5af1ac2f92daf8986ac2ae01de0992e092587af1dc5"}
Jul 09 18:04:56.628501 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.628474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:56.655504 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:56.655457    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggx9d" podStartSLOduration=4.0095311 podStartE2EDuration="27.65544265s" totalImagesPullingTime="23.64591155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:56.655281551 +0000 UTC m=+27.756808578" watchObservedRunningTime="2026-07-09 18:04:56.65544265 +0000 UTC m=+27.756969721"
Jul 09 18:04:57.449482 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.449397    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:57.449698 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:57.449512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:57.623381 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.623358    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:04:57.623811 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.623689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" event={"ID":"fb87d44a-492c-434b-8dd9-23798314b545","Type":"ContainerStarted","Data":"07a861938b6b7fb79d8e84a92509cff3ecd9b609c47cef6debf2e371e8a79621"}
Jul 09 18:04:57.624026 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.624011    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:57.624091 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.624077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:57.638928 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.638901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:04:57.650375 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:57.650331    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr" podStartSLOduration=9.77589154 podStartE2EDuration="28.650318038s" totalImagesPullingTime="18.874426498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:04:57.649259878 +0000 UTC m=+28.750786905" watchObservedRunningTime="2026-07-09 18:04:57.650318038 +0000 UTC m=+28.751845064"
Jul 09 18:04:58.449769 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.449738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:58.449983 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.449784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:58.449983 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:58.449851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:58.450057 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:58.449972    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:04:58.496518 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.496481    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zfwnv"]
Jul 09 18:04:58.496698 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.496592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:04:58.496778 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:58.496712    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:04:58.497168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.497150    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dwcb"]
Jul 09 18:04:58.498421 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.498400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blwkj"]
Jul 09 18:04:58.628019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.627936    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="f34aadb35b50ec982a857bc36bdbb5d9d1f6b389b46cf5840bf394a983a0fb18" exitCode=0
Jul 09 18:04:58.628445 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.628020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:04:58.628445 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.628027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"f34aadb35b50ec982a857bc36bdbb5d9d1f6b389b46cf5840bf394a983a0fb18"}
Jul 09 18:04:58.628445 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:04:58.628104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:04:58.628445 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:58.628115    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:04:58.628445 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:04:58.628176    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:05:00.449140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.448956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:00.449550 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.448982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:05:00.449550 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:00.449213    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:05:00.449550 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.448957    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:00.449550 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:00.449302    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:05:00.449550 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:00.449406    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:05:00.633927 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.633842    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="6f4ce220f1b95ff315b70d23d46fb480fc05df8ee3937fa88f49788da6da8f44" exitCode=0
Jul 09 18:05:00.633927 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.633909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"6f4ce220f1b95ff315b70d23d46fb480fc05df8ee3937fa88f49788da6da8f44"}
Jul 09 18:05:00.911377 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.911289    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:05:00.911538 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.911496    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:05:00.911954 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:00.911930    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xrz68"
Jul 09 18:05:02.449535 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:02.449454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:02.450106 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:02.449454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:05:02.450106 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:02.449594    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2dwcb" podUID="8b2b359c-3de6-404d-b061-b0fe458d8a4a"
Jul 09 18:05:02.450106 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:02.449454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:02.450106 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:02.449675    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:05:02.450106 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:02.449754    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zfwnv" podUID="c451f403-441f-4d69-992a-e55962af5083"
Jul 09 18:05:03.037555 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:03.037513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:03.037555 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:03.037560    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:05:03.037831 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.037671    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:03.037831 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.037676    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:03.037831 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.037748    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:35.037713108 +0000 UTC m=+66.139240133 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:05:03.037831 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.037763    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret podName:c451f403-441f-4d69-992a-e55962af5083 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:35.037756849 +0000 UTC m=+66.139283854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret") pod "global-pull-secret-syncer-zfwnv" (UID: "c451f403-441f-4d69-992a-e55962af5083") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:05:03.138512 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:03.138482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:03.138708 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.138677    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:05:03.138708 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.138704    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:05:03.138859 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.138718    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nnzth for pod openshift-network-diagnostics/network-check-target-2dwcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:03.138859 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:03.138805    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth podName:8b2b359c-3de6-404d-b061-b0fe458d8a4a nodeName:}" failed. No retries permitted until 2026-07-09 18:05:35.138785016 +0000 UTC m=+66.240312038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnzth" (UniqueName: "kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth") pod "network-check-target-2dwcb" (UID: "8b2b359c-3de6-404d-b061-b0fe458d8a4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:05:04.166949 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.166913    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-20.ec2.internal" event="NodeReady"
Jul 09 18:05:04.167513 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.167052    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:05:04.229612 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.229575    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7mkq2"]
Jul 09 18:05:04.234943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.234909    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5rmsk"]
Jul 09 18:05:04.235119 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.235062    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.236623 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.236600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.238102 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.238084    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:05:04.238239 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.238214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:05:04.238332 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.238300    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jd75b\""
Jul 09 18:05:04.239848 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.239829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:05:04.240086 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.240069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:05:04.240214 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.240195    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:05:04.240431 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.240412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lkb9p\""
Jul 09 18:05:04.247414 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.247232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mkq2"]
Jul 09 18:05:04.247781 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.247759    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5rmsk"]
Jul 09 18:05:04.348899 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.348854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b0b17-5d17-4327-87d0-f5904cb3aa04-config-volume\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.348899 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.348905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b0b17-5d17-4327-87d0-f5904cb3aa04-tmp-dir\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.349137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.348938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pch5\" (UniqueName: \"kubernetes.io/projected/d56b0b17-5d17-4327-87d0-f5904cb3aa04-kube-api-access-7pch5\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.349137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.349034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.349137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.349061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648fd878-2053-4126-b5b7-6bad43fe0141-cert\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.349137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.349095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5759\" (UniqueName: \"kubernetes.io/projected/648fd878-2053-4126-b5b7-6bad43fe0141-kube-api-access-q5759\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.449202 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:04.449202 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449147    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:04.449202 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:05:04.449477 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pch5\" (UniqueName: \"kubernetes.io/projected/d56b0b17-5d17-4327-87d0-f5904cb3aa04-kube-api-access-7pch5\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.449567 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.449635 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648fd878-2053-4126-b5b7-6bad43fe0141-cert\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.449635 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5759\" (UniqueName: \"kubernetes.io/projected/648fd878-2053-4126-b5b7-6bad43fe0141-kube-api-access-q5759\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.449635 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b0b17-5d17-4327-87d0-f5904cb3aa04-config-volume\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.449805 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.449652    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b0b17-5d17-4327-87d0-f5904cb3aa04-tmp-dir\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.449805 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:04.449656    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:04.449805 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:04.449748    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:04.949706173 +0000 UTC m=+36.051233202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:04.450162 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.450131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d56b0b17-5d17-4327-87d0-f5904cb3aa04-tmp-dir\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.450354 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.450287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d56b0b17-5d17-4327-87d0-f5904cb3aa04-config-volume\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.454776 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.454751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/648fd878-2053-4126-b5b7-6bad43fe0141-cert\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.458658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.458461    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:04.458658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.458523    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w9bx9\""
Jul 09 18:05:04.458658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.458612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:04.459234 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.459135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:04.459415 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.459396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:04.459959 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.459860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k8ns\""
Jul 09 18:05:04.469472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.469174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pch5\" (UniqueName: \"kubernetes.io/projected/d56b0b17-5d17-4327-87d0-f5904cb3aa04-kube-api-access-7pch5\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.469472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.469258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5759\" (UniqueName: \"kubernetes.io/projected/648fd878-2053-4126-b5b7-6bad43fe0141-kube-api-access-q5759\") pod \"ingress-canary-5rmsk\" (UID: \"648fd878-2053-4126-b5b7-6bad43fe0141\") " pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.553835 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.553801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5rmsk"
Jul 09 18:05:04.955127 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:04.955091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:04.955277 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:04.955224    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:04.955349 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:04.955278    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:05.955265282 +0000 UTC m=+37.056792287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:05.154234 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:05.154056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5rmsk"]
Jul 09 18:05:05.158677 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:05:05.158658    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648fd878_2053_4126_b5b7_6bad43fe0141.slice/crio-822edace242561fbbefbb4102ce39c7191f66f693e0fd3466a55e6b7eb265ce4 WatchSource:0}: Error finding container 822edace242561fbbefbb4102ce39c7191f66f693e0fd3466a55e6b7eb265ce4: Status 404 returned error can't find the container with id 822edace242561fbbefbb4102ce39c7191f66f693e0fd3466a55e6b7eb265ce4
Jul 09 18:05:05.648987 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:05.648959    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="f62d693f52eb709788bba8f9e6b9027917cb467597d21351d93a0a71b39b1dbe" exitCode=0
Jul 09 18:05:05.649670 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:05.649039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"f62d693f52eb709788bba8f9e6b9027917cb467597d21351d93a0a71b39b1dbe"}
Jul 09 18:05:05.650508 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:05.650485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5rmsk" event={"ID":"648fd878-2053-4126-b5b7-6bad43fe0141","Type":"ContainerStarted","Data":"822edace242561fbbefbb4102ce39c7191f66f693e0fd3466a55e6b7eb265ce4"}
Jul 09 18:05:05.964704 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:05.964603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:05.964920 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:05.964790    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:05.964920 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:05.964872    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:07.964851993 +0000 UTC m=+39.066379013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:06.655679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:06.655646    2561 generic.go:356] "Generic (PLEG): container finished" podID="5149ade1-96ea-4336-90a2-3b0f53c2e8da" containerID="8b852395993bb72585d9033a0e37f0130136c7c8f8daa23b671ef0cd67a4afab" exitCode=0
Jul 09 18:05:06.656151 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:06.655684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerDied","Data":"8b852395993bb72585d9033a0e37f0130136c7c8f8daa23b671ef0cd67a4afab"}
Jul 09 18:05:07.662395 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:07.662360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" event={"ID":"5149ade1-96ea-4336-90a2-3b0f53c2e8da","Type":"ContainerStarted","Data":"3605b974a13019942b96fc42447676bdf458295d34de708ab24cf35dfd549c00"}
Jul 09 18:05:07.693094 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:07.693036    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k7vpg" podStartSLOduration=8.252901671 podStartE2EDuration="38.693014478s" totalImagesPullingTime="30.440112807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:07.692715472 +0000 UTC m=+38.794242499" watchObservedRunningTime="2026-07-09 18:05:07.693014478 +0000 UTC m=+38.794541505"
Jul 09 18:05:07.983521 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:07.983433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:07.983659 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:07.983577    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:07.983659 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:07.983643    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:11.983628017 +0000 UTC m=+43.085155022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:08.666551 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:08.666299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5rmsk" event={"ID":"648fd878-2053-4126-b5b7-6bad43fe0141","Type":"ContainerStarted","Data":"db8a6b07433a89647617a0878515f73c2471e0afaaae17fd819e27af7edf8722"}
Jul 09 18:05:08.681792 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:08.681687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5rmsk" podStartSLOduration=1.782999744 podStartE2EDuration="4.681668787s" totalImagesPullingTime="2.898669043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:08.681330725 +0000 UTC m=+39.782857738" watchObservedRunningTime="2026-07-09 18:05:08.681668787 +0000 UTC m=+39.783195814"
Jul 09 18:05:12.011411 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:12.011373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:12.011800 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:12.011487    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:12.011800 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:12.011536    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:20.01152385 +0000 UTC m=+51.113050854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:17.558769 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.558718    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"]
Jul 09 18:05:17.590136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.590103    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"]
Jul 09 18:05:17.590281 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.590211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.593962 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.593941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:05:17.594773 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.594754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:05:17.594773 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.594766    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6hnbx\""
Jul 09 18:05:17.594894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.594777    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:05:17.594894 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.594815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:05:17.651596 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.651567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r262c\" (UniqueName: \"kubernetes.io/projected/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-kube-api-access-r262c\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.651770 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.651603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.752056 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.752016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r262c\" (UniqueName: \"kubernetes.io/projected/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-kube-api-access-r262c\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.752056 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.752059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.756217 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.756196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.761175 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.761152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r262c\" (UniqueName: \"kubernetes.io/projected/59d6bcc8-7254-4021-b5a2-ec3d03b87ec6-kube-api-access-r262c\") pod \"managed-serviceaccount-addon-agent-854897756d-dtkpc\" (UID: \"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:17.910438 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:17.910350    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"
Jul 09 18:05:18.045942 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:18.045909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc"]
Jul 09 18:05:18.049667 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:05:18.049641    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59d6bcc8_7254_4021_b5a2_ec3d03b87ec6.slice/crio-8aa984017b5b0e1425f40fc2d66fbeada34c08dd7ef5e8d105d91c952432a159 WatchSource:0}: Error finding container 8aa984017b5b0e1425f40fc2d66fbeada34c08dd7ef5e8d105d91c952432a159: Status 404 returned error can't find the container with id 8aa984017b5b0e1425f40fc2d66fbeada34c08dd7ef5e8d105d91c952432a159
Jul 09 18:05:18.688666 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:18.688626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc" event={"ID":"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6","Type":"ContainerStarted","Data":"8aa984017b5b0e1425f40fc2d66fbeada34c08dd7ef5e8d105d91c952432a159"}
Jul 09 18:05:20.069385 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:20.069347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:20.069812 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:20.069527    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:20.069812 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:20.069610    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:05:36.06958787 +0000 UTC m=+67.171114891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:21.696770 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:21.696733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc" event={"ID":"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6","Type":"ContainerStarted","Data":"2cfaceb8e56d6c6ee62422fe92d6a954c4fb62a417e51e8aa07716eee21e0df0"}
Jul 09 18:05:21.714061 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:21.714016    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc" podStartSLOduration=1.664098539 podStartE2EDuration="4.714001358s" totalImagesPullingTime="3.049902819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:21.713239784 +0000 UTC m=+52.814766810" watchObservedRunningTime="2026-07-09 18:05:21.714001358 +0000 UTC m=+52.815528385"
Jul 09 18:05:29.642462 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:29.642435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5t7cr"
Jul 09 18:05:35.070637 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.070596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:35.070637 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.070639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:05:35.073278 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.073260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:05:35.073342 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.073294    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:05:35.081245 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:35.081221    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:05:35.081366 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:35.081282    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:39.081265346 +0000 UTC m=+130.182792364 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : secret "metrics-daemon-secret" not found
Jul 09 18:05:35.083206 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.083187    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c451f403-441f-4d69-992a-e55962af5083-original-pull-secret\") pod \"global-pull-secret-syncer-zfwnv\" (UID: \"c451f403-441f-4d69-992a-e55962af5083\") " pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:35.171766 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.171737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:35.174022 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.174004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:05:35.184049 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.184032    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:05:35.194241 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.194221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzth\" (UniqueName: \"kubernetes.io/projected/8b2b359c-3de6-404d-b061-b0fe458d8a4a-kube-api-access-nnzth\") pod \"network-check-target-2dwcb\" (UID: \"8b2b359c-3de6-404d-b061-b0fe458d8a4a\") " pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:35.368691 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.368617    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w9bx9\""
Jul 09 18:05:35.375810 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.375791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:35.378089 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.378074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zfwnv"
Jul 09 18:05:35.506506 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.506475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2dwcb"]
Jul 09 18:05:35.509553 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:05:35.509525    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2b359c_3de6_404d_b061_b0fe458d8a4a.slice/crio-4f44f1118572bbbf8fa58f51638956da62efba72812adb606985d0da1a67a04f WatchSource:0}: Error finding container 4f44f1118572bbbf8fa58f51638956da62efba72812adb606985d0da1a67a04f: Status 404 returned error can't find the container with id 4f44f1118572bbbf8fa58f51638956da62efba72812adb606985d0da1a67a04f
Jul 09 18:05:35.520651 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.520629    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zfwnv"]
Jul 09 18:05:35.523448 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:05:35.523427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc451f403_441f_4d69_992a_e55962af5083.slice/crio-0522a2938108910c99b9286410075ac6a35362059d616ea64ca35ee5cb3b4873 WatchSource:0}: Error finding container 0522a2938108910c99b9286410075ac6a35362059d616ea64ca35ee5cb3b4873: Status 404 returned error can't find the container with id 0522a2938108910c99b9286410075ac6a35362059d616ea64ca35ee5cb3b4873
Jul 09 18:05:35.729927 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.729840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zfwnv" event={"ID":"c451f403-441f-4d69-992a-e55962af5083","Type":"ContainerStarted","Data":"0522a2938108910c99b9286410075ac6a35362059d616ea64ca35ee5cb3b4873"}
Jul 09 18:05:35.730975 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:35.730952    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dwcb" event={"ID":"8b2b359c-3de6-404d-b061-b0fe458d8a4a","Type":"ContainerStarted","Data":"4f44f1118572bbbf8fa58f51638956da62efba72812adb606985d0da1a67a04f"}
Jul 09 18:05:36.080342 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:36.080309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:05:36.080797 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:36.080446    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:05:36.080797 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:05:36.080546    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:06:08.080526753 +0000 UTC m=+99.182053762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:05:40.748747 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:40.748625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zfwnv" event={"ID":"c451f403-441f-4d69-992a-e55962af5083","Type":"ContainerStarted","Data":"3488ec135d9084c8369ad6e1fe4e09e37736dd23f4da4b72ea95aa4237200f7b"}
Jul 09 18:05:40.750363 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:40.750325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2dwcb" event={"ID":"8b2b359c-3de6-404d-b061-b0fe458d8a4a","Type":"ContainerStarted","Data":"aa8b9fd4da9105170d008e21e90bcbef09150c2ae7801fcbad18d31b652f3097"}
Jul 09 18:05:40.752753 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:40.750771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:05:40.765181 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:40.765140    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zfwnv" podStartSLOduration=66.977592016 podStartE2EDuration="1m11.765125841s" totalImagesPullingTime="4.787533825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:40.764489447 +0000 UTC m=+71.866016475" watchObservedRunningTime="2026-07-09 18:05:40.765125841 +0000 UTC m=+71.866652867"
Jul 09 18:05:40.778243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:05:40.778184    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2dwcb" podStartSLOduration=66.977755969 podStartE2EDuration="1m11.7781688s" totalImagesPullingTime="4.800412831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:40.777926156 +0000 UTC m=+71.879453184" watchObservedRunningTime="2026-07-09 18:05:40.7781688 +0000 UTC m=+71.879695828"
Jul 09 18:06:08.107403 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:06:08.107361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:06:08.107816 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:06:08.107486    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:06:08.107816 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:06:08.107551    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:07:12.107534972 +0000 UTC m=+163.209061977 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:06:11.756130 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:06:11.756102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2dwcb"
Jul 09 18:06:39.123688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:06:39.123638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:06:39.124109 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:06:39.123816    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 18:06:39.124109 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:06:39.123894    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs podName:c5f59614-c402-4162-87a3-ab09262ca067 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:41.123878396 +0000 UTC m=+252.225405401 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs") pod "network-metrics-daemon-blwkj" (UID: "c5f59614-c402-4162-87a3-ab09262ca067") : secret "metrics-daemon-secret" not found
Jul 09 18:07:07.248955 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:07:07.248909    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7mkq2" podUID="d56b0b17-5d17-4327-87d0-f5904cb3aa04"
Jul 09 18:07:07.483566 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:07:07.483510    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-blwkj" podUID="c5f59614-c402-4162-87a3-ab09262ca067"
Jul 09 18:07:07.963273 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:07.963247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:07:12.148106 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:12.148067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:07:12.148793 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:07:12.148770    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:07:12.148990 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:07:12.148977    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls podName:d56b0b17-5d17-4327-87d0-f5904cb3aa04 nodeName:}" failed. No retries permitted until 2026-07-09 18:09:14.148946511 +0000 UTC m=+285.250473531 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls") pod "dns-default-7mkq2" (UID: "d56b0b17-5d17-4327-87d0-f5904cb3aa04") : secret "dns-default-metrics-tls" not found
Jul 09 18:07:19.449926 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:19.449889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:07:21.999029 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:21.998995    2561 generic.go:356] "Generic (PLEG): container finished" podID="59d6bcc8-7254-4021-b5a2-ec3d03b87ec6" containerID="2cfaceb8e56d6c6ee62422fe92d6a954c4fb62a417e51e8aa07716eee21e0df0" exitCode=255
Jul 09 18:07:21.999406 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:21.999057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc" event={"ID":"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6","Type":"ContainerDied","Data":"2cfaceb8e56d6c6ee62422fe92d6a954c4fb62a417e51e8aa07716eee21e0df0"}
Jul 09 18:07:22.004615 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:22.004594    2561 scope.go:117] "RemoveContainer" containerID="2cfaceb8e56d6c6ee62422fe92d6a954c4fb62a417e51e8aa07716eee21e0df0"
Jul 09 18:07:23.006473 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:07:23.006442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-854897756d-dtkpc" event={"ID":"59d6bcc8-7254-4021-b5a2-ec3d03b87ec6","Type":"ContainerStarted","Data":"111a3ec71cd78aada01cc68117988242ba3eb1d372519e50173194f859b4dfc9"}
Jul 09 18:08:08.642475 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.642439    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"]
Jul 09 18:08:08.644453 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.644437    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f565474d4-kwrxx"]
Jul 09 18:08:08.644573 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.644555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"
Jul 09 18:08:08.646214 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.646197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.651017 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.650989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 18:08:08.651017 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.650992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 18:08:08.651184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651072    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 18:08:08.651184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 18:08:08.651590 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pg6hn\""
Jul 09 18:08:08.651677 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6v2xm\""
Jul 09 18:08:08.651677 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651642    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 18:08:08.651677 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.651577    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 18:08:08.655300 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.655281    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"]
Jul 09 18:08:08.660350 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.660328    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f565474d4-kwrxx"]
Jul 09 18:08:08.749203 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749171    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-stats-auth\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.749203 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.749472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-default-certificate\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.749472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749324    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmtfq\" (UniqueName: \"kubernetes.io/projected/59c8f550-78fc-4f3d-aede-39473570c0cd-kube-api-access-qmtfq\") pod \"network-check-source-869dcfcdcd-qknbm\" (UID: \"59c8f550-78fc-4f3d-aede-39473570c0cd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"
Jul 09 18:08:08.749472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.749472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.749410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfz4h\" (UniqueName: \"kubernetes.io/projected/ac28916c-868d-41f0-bfbf-b1ec473d8903-kube-api-access-sfz4h\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.760046 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.760010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"]
Jul 09 18:08:08.762112 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.762096    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:08:08.762236 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.762223    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.763836 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.763818    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.765995 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.765980    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 18:08:08.767362 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.767347    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:08:08.768489 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.768335    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:08:08.770302 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.770286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bsf5x\""
Jul 09 18:08:08.770613 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.770595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 18:08:08.776236 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.776219    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:08:08.776479 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.776459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6mm8x\""
Jul 09 18:08:08.776479 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.776475    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:08:08.776618 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.776464    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 18:08:08.781826 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.781803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"]
Jul 09 18:08:08.782690 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.782666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:08:08.783602 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.783578    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:08:08.836336 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.836300    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-jxn6g"]
Jul 09 18:08:08.838520 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.838500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:08.842441 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.842416    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:08:08.843086 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.843029    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 18:08:08.843370 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.843356    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5pxzx\""
Jul 09 18:08:08.843940 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.843922    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 18:08:08.843940 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.843934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 18:08:08.847320 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.847299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 18:08:08.849970 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.849951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jxn6g"]
Jul 09 18:08:08.850184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.850234 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.850271 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfbm\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.850271 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-config\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.850358 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.850358 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.850307    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:09.350287747 +0000 UTC m=+220.451814752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:08.850460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.850460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdkr\" (UniqueName: \"kubernetes.io/projected/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-kube-api-access-gvdkr\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.850460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850440    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfz4h\" (UniqueName: \"kubernetes.io/projected/ac28916c-868d-41f0-bfbf-b1ec473d8903-kube-api-access-sfz4h\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850475    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-stats-auth\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-default-certificate\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.850639    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.850682    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:09.350667369 +0000 UTC m=+220.452194377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : secret "router-metrics-certs-default" not found
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmtfq\" (UniqueName: \"kubernetes.io/projected/59c8f550-78fc-4f3d-aede-39473570c0cd-kube-api-access-qmtfq\") pod \"network-check-source-869dcfcdcd-qknbm\" (UID: \"59c8f550-78fc-4f3d-aede-39473570c0cd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"
Jul 09 18:08:08.850797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.851143 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.851143 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.851143 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.850875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.853222 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.853191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-stats-auth\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.853323 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.853253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-default-certificate\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.863063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.863041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfz4h\" (UniqueName: \"kubernetes.io/projected/ac28916c-868d-41f0-bfbf-b1ec473d8903-kube-api-access-sfz4h\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:08.865300 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.865277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmtfq\" (UniqueName: \"kubernetes.io/projected/59c8f550-78fc-4f3d-aede-39473570c0cd-kube-api-access-qmtfq\") pod \"network-check-source-869dcfcdcd-qknbm\" (UID: \"59c8f550-78fc-4f3d-aede-39473570c0cd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"
Jul 09 18:08:08.952047 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.951955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952047 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.951991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952047 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952293 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-config\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.952293 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952293 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952438 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952438 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.952413    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:08:08.952438 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.952427    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c978d5698-jzqrs: secret "image-registry-tls" not found
Jul 09 18:08:08.952573 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:08.952493    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls podName:d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:09.452458724 +0000 UTC m=+220.553985744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls") pod "image-registry-7c978d5698-jzqrs" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9") : secret "image-registry-tls" not found
Jul 09 18:08:08.952573 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.952682 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngfpq\" (UniqueName: \"kubernetes.io/projected/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-kube-api-access-ngfpq\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:08.952682 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-config\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:08.952682 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952887 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-trusted-ca\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:08.952887 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfbm\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.952887 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-config\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.952887 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdkr\" (UniqueName: \"kubernetes.io/projected/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-kube-api-access-gvdkr\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.952887 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-serving-cert\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:08.953136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.953136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.952991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.953136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.953025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.954528 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.954505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.954753 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.954737    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-serving-cert\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.954911 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.954897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.958344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.958329    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"
Jul 09 18:08:08.963937 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.963919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:08.965758 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.965705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdkr\" (UniqueName: \"kubernetes.io/projected/fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9-kube-api-access-gvdkr\") pod \"service-ca-operator-7695fc6c8c-5pm5d\" (UID: \"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:08.966362 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:08.966341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfbm\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:09.054222 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.054186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngfpq\" (UniqueName: \"kubernetes.io/projected/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-kube-api-access-ngfpq\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.054222 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.054223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-config\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.054418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.054256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-trusted-ca\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.054465 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.054422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-serving-cert\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.055015 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.054989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-config\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.055245 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.055226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-trusted-ca\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.056755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.056712    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-serving-cert\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.063353 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.063329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngfpq\" (UniqueName: \"kubernetes.io/projected/00a0f1dd-da67-4730-9820-8fc0ec8ac69e-kube-api-access-ngfpq\") pod \"console-operator-7866497494-jxn6g\" (UID: \"00a0f1dd-da67-4730-9820-8fc0ec8ac69e\") " pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.072158 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.072138    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"
Jul 09 18:08:09.077816 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.077792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm"]
Jul 09 18:08:09.080354 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:09.080325    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8f550_78fc_4f3d_aede_39473570c0cd.slice/crio-c0f2b6ec171434ab72b5a02e5c39758d822802b856000d122521d40cc29fbc61 WatchSource:0}: Error finding container c0f2b6ec171434ab72b5a02e5c39758d822802b856000d122521d40cc29fbc61: Status 404 returned error can't find the container with id c0f2b6ec171434ab72b5a02e5c39758d822802b856000d122521d40cc29fbc61
Jul 09 18:08:09.139349 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.139321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm" event={"ID":"59c8f550-78fc-4f3d-aede-39473570c0cd","Type":"ContainerStarted","Data":"3a1e623a6126af0a40b955161081afeceef26adff3c4074396d9fb0e8a7bddcd"}
Jul 09 18:08:09.139479 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.139371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm" event={"ID":"59c8f550-78fc-4f3d-aede-39473570c0cd","Type":"ContainerStarted","Data":"c0f2b6ec171434ab72b5a02e5c39758d822802b856000d122521d40cc29fbc61"}
Jul 09 18:08:09.147329 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.147300    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:09.154361 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.154123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-qknbm" podStartSLOduration=1.154101407 podStartE2EDuration="1.154101407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:09.153075021 +0000 UTC m=+220.254602074" watchObservedRunningTime="2026-07-09 18:08:09.154101407 +0000 UTC m=+220.255628437"
Jul 09 18:08:09.205876 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.205753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d"]
Jul 09 18:08:09.212245 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:09.212206    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb263d2_3721_4f5c_bbda_fb44c9d4b1e9.slice/crio-08596873789f5646bfc96e3647dc6d216614e4c07ab4a3e4aecfb6606f482cee WatchSource:0}: Error finding container 08596873789f5646bfc96e3647dc6d216614e4c07ab4a3e4aecfb6606f482cee: Status 404 returned error can't find the container with id 08596873789f5646bfc96e3647dc6d216614e4c07ab4a3e4aecfb6606f482cee
Jul 09 18:08:09.286608 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.286564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jxn6g"]
Jul 09 18:08:09.290560 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:09.290536    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a0f1dd_da67_4730_9820_8fc0ec8ac69e.slice/crio-29a6ce51e86bd0a7a14730978e08cfbb6490190978a0bf5ee7a97b891db7fcff WatchSource:0}: Error finding container 29a6ce51e86bd0a7a14730978e08cfbb6490190978a0bf5ee7a97b891db7fcff: Status 404 returned error can't find the container with id 29a6ce51e86bd0a7a14730978e08cfbb6490190978a0bf5ee7a97b891db7fcff
Jul 09 18:08:09.356434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.356399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:09.356600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.356508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:09.356600 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.356551    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:08:09.356710 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.356615    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:10.356599906 +0000 UTC m=+221.458126911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : secret "router-metrics-certs-default" not found
Jul 09 18:08:09.356710 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.356631    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:10.356624014 +0000 UTC m=+221.458151019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:09.457718 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:09.457635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:09.457875 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.457759    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:08:09.457875 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.457771    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c978d5698-jzqrs: secret "image-registry-tls" not found
Jul 09 18:08:09.457875 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:09.457818    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls podName:d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:10.457804534 +0000 UTC m=+221.559331540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls") pod "image-registry-7c978d5698-jzqrs" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9") : secret "image-registry-tls" not found
Jul 09 18:08:10.144387 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:10.144329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d" event={"ID":"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9","Type":"ContainerStarted","Data":"08596873789f5646bfc96e3647dc6d216614e4c07ab4a3e4aecfb6606f482cee"}
Jul 09 18:08:10.145709 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:10.145677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jxn6g" event={"ID":"00a0f1dd-da67-4730-9820-8fc0ec8ac69e","Type":"ContainerStarted","Data":"29a6ce51e86bd0a7a14730978e08cfbb6490190978a0bf5ee7a97b891db7fcff"}
Jul 09 18:08:10.365699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:10.365658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:10.365914 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:10.365802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:10.365914 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.365851    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:08:10.366033 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.365926    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:12.365908169 +0000 UTC m=+223.467435179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : secret "router-metrics-certs-default" not found
Jul 09 18:08:10.366033 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.365941    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:12.365935471 +0000 UTC m=+223.467462476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:10.467144 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:10.467065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:10.467289 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.467232    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:08:10.467289 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.467255    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c978d5698-jzqrs: secret "image-registry-tls" not found
Jul 09 18:08:10.467382 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:10.467312    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls podName:d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:12.467297399 +0000 UTC m=+223.568824403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls") pod "image-registry-7c978d5698-jzqrs" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9") : secret "image-registry-tls" not found
Jul 09 18:08:12.153098 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.153058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d" event={"ID":"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9","Type":"ContainerStarted","Data":"36dd204a6a8c4b3c4caa9f3ff0d19ac4e2cf03eefef0624b30f34afe7a4cdcda"}
Jul 09 18:08:12.154707 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.154684    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/0.log"
Jul 09 18:08:12.154838 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.154744    2561 generic.go:356] "Generic (PLEG): container finished" podID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e" containerID="99735d3895f2ca65e0729f934a4b11870343364af24e7fe004bf24c97d930054" exitCode=255
Jul 09 18:08:12.154838 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.154783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jxn6g" event={"ID":"00a0f1dd-da67-4730-9820-8fc0ec8ac69e","Type":"ContainerDied","Data":"99735d3895f2ca65e0729f934a4b11870343364af24e7fe004bf24c97d930054"}
Jul 09 18:08:12.155015 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.154999    2561 scope.go:117] "RemoveContainer" containerID="99735d3895f2ca65e0729f934a4b11870343364af24e7fe004bf24c97d930054"
Jul 09 18:08:12.173490 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.173433    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d" podStartSLOduration=1.8620767580000002 podStartE2EDuration="4.17340509s" totalImagesPullingTime="2.311328332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:12.171879633 +0000 UTC m=+223.273406661" watchObservedRunningTime="2026-07-09 18:08:12.17340509 +0000 UTC m=+223.274932118"
Jul 09 18:08:12.383806 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.383766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:12.383946 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.383824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:12.383946 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.383924    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:16.383908007 +0000 UTC m=+227.485435013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:12.383946 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.383944    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:08:12.384070 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.384019    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:16.384003694 +0000 UTC m=+227.485530716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : secret "router-metrics-certs-default" not found
Jul 09 18:08:12.485046 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:12.484637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:12.485046 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.484893    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:08:12.485046 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.484910    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c978d5698-jzqrs: secret "image-registry-tls" not found
Jul 09 18:08:12.485046 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:12.484963    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls podName:d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:16.484945474 +0000 UTC m=+227.586472485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls") pod "image-registry-7c978d5698-jzqrs" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9") : secret "image-registry-tls" not found
Jul 09 18:08:13.162644 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.162611    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/1.log"
Jul 09 18:08:13.163063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.162988    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/0.log"
Jul 09 18:08:13.163063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.163017    2561 generic.go:356] "Generic (PLEG): container finished" podID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f" exitCode=255
Jul 09 18:08:13.163161 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.163107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jxn6g" event={"ID":"00a0f1dd-da67-4730-9820-8fc0ec8ac69e","Type":"ContainerDied","Data":"cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"}
Jul 09 18:08:13.163218 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.163192    2561 scope.go:117] "RemoveContainer" containerID="99735d3895f2ca65e0729f934a4b11870343364af24e7fe004bf24c97d930054"
Jul 09 18:08:13.163414 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.163392    2561 scope.go:117] "RemoveContainer" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"
Jul 09 18:08:13.163609 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:13.163588    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:13.378469 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.378434    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nqprt"]
Jul 09 18:08:13.381121 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.381097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.385528 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.385507    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:08:13.385642 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.385599    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:08:13.386328 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.386310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:08:13.386328 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.386323    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:08:13.386582 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.386568    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lw2dl\""
Jul 09 18:08:13.399483 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.399463    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nqprt"]
Jul 09 18:08:13.494315 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.494225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a23042-fd7e-4bce-8ba9-d760919382ec-crio-socket\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.494315 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.494275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.494315 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.494305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.494553 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.494335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a23042-fd7e-4bce-8ba9-d760919382ec-data-volume\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.494553 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.494363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6pd\" (UniqueName: \"kubernetes.io/projected/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-api-access-zr6pd\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595312 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a23042-fd7e-4bce-8ba9-d760919382ec-crio-socket\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:13.595412    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:13.595440    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:13.595473    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:14.095459664 +0000 UTC m=+225.196986669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44a23042-fd7e-4bce-8ba9-d760919382ec-crio-socket\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595500 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:13.595495    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:14.095481087 +0000 UTC m=+225.197008092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:13.595892 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a23042-fd7e-4bce-8ba9-d760919382ec-data-volume\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595892 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6pd\" (UniqueName: \"kubernetes.io/projected/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-api-access-zr6pd\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.595892 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.595815    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44a23042-fd7e-4bce-8ba9-d760919382ec-data-volume\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:13.604083 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:13.604064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6pd\" (UniqueName: \"kubernetes.io/projected/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-api-access-zr6pd\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:14.099879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:14.099840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:14.099879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:14.099879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:14.100196 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:14.099960    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:14.100196 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:14.100005    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:14.100196 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:14.100026    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:15.100008646 +0000 UTC m=+226.201535672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:14.100196 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:14.100069    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:15.100050574 +0000 UTC m=+226.201577578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:14.167830 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:14.167804    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/1.log"
Jul 09 18:08:14.168167 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:14.168128    2561 scope.go:117] "RemoveContainer" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"
Jul 09 18:08:14.168301 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:14.168281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:15.109385 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:15.109346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:15.109385 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:15.109384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:15.109600 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:15.109472    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:15.109600 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:15.109505    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:15.109600 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:15.109531    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:17.109514788 +0000 UTC m=+228.211041814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:15.109600 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:15.109557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:17.109544616 +0000 UTC m=+228.211071624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:15.572024 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:15.571995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vk8gc_c2afda61-2c05-44c1-a4ff-c73ef0b46e2d/dns-node-resolver/0.log"
Jul 09 18:08:16.420530 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:16.420478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:16.420530 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:16.420544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:16.420840 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.420685    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:08:16.420840 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.420685    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:24.420661603 +0000 UTC m=+235.522188614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:16.420840 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.420756    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:24.420741406 +0000 UTC m=+235.522268412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : secret "router-metrics-certs-default" not found
Jul 09 18:08:16.520968 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:16.520934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:16.521134 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.521086    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:08:16.521134 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.521106    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c978d5698-jzqrs: secret "image-registry-tls" not found
Jul 09 18:08:16.521200 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:16.521159    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls podName:d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:24.521144923 +0000 UTC m=+235.622671929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls") pod "image-registry-7c978d5698-jzqrs" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9") : secret "image-registry-tls" not found
Jul 09 18:08:16.571548 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:16.571522    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szr88_87db40f4-a668-401e-9eb0-737aba2f6a03/node-ca/0.log"
Jul 09 18:08:16.972807 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:16.972775    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5rmsk_648fd878-2053-4126-b5b7-6bad43fe0141/serve-healthcheck-canary/0.log"
Jul 09 18:08:17.125882 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:17.125843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:17.125882 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:17.125884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:17.126091 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:17.125987    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:17.126091 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:17.126040    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:21.126026871 +0000 UTC m=+232.227553875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:17.126091 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:17.125994    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:17.126213 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:17.126111    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:21.126099552 +0000 UTC m=+232.227626557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:19.147653 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:19.147619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:19.148036 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:19.147695    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:19.148036 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:19.148003    2561 scope.go:117] "RemoveContainer" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"
Jul 09 18:08:19.148146 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:19.148126    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:21.163622 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:21.163590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:21.163622 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:21.163628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:21.164113 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:21.163708    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:08:21.164113 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:21.163787    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:21.164113 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:21.163792    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:29.16377726 +0000 UTC m=+240.265304264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : configmap "kube-rbac-proxy" not found
Jul 09 18:08:21.164113 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:21.163881    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls podName:44a23042-fd7e-4bce-8ba9-d760919382ec nodeName:}" failed. No retries permitted until 2026-07-09 18:08:29.163862342 +0000 UTC m=+240.265389346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nqprt" (UID: "44a23042-fd7e-4bce-8ba9-d760919382ec") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:08:24.490978 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.490937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:24.491364 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.491000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:24.491364 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:24.491046    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle podName:ac28916c-868d-41f0-bfbf-b1ec473d8903 nodeName:}" failed. No retries permitted until 2026-07-09 18:08:40.491028529 +0000 UTC m=+251.592555535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle") pod "router-default-6f565474d4-kwrxx" (UID: "ac28916c-868d-41f0-bfbf-b1ec473d8903") : configmap references non-existent config key: service-ca.crt
Jul 09 18:08:24.493344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.493319    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac28916c-868d-41f0-bfbf-b1ec473d8903-metrics-certs\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:24.591479 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.591439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:24.593795 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.593774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"image-registry-7c978d5698-jzqrs\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") " pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:24.677264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.677214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:24.802629 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:24.802553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:08:24.806048 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:24.806024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bc18ba_74d1_4392_ad9e_ab3f4c659cc9.slice/crio-b51ddac51d0c6e662685482192fed9d9cfb0af54b4b1f50d5be9186d16f6b06a WatchSource:0}: Error finding container b51ddac51d0c6e662685482192fed9d9cfb0af54b4b1f50d5be9186d16f6b06a: Status 404 returned error can't find the container with id b51ddac51d0c6e662685482192fed9d9cfb0af54b4b1f50d5be9186d16f6b06a
Jul 09 18:08:25.199821 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:25.199789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" event={"ID":"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9","Type":"ContainerStarted","Data":"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"}
Jul 09 18:08:25.199981 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:25.199860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" event={"ID":"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9","Type":"ContainerStarted","Data":"b51ddac51d0c6e662685482192fed9d9cfb0af54b4b1f50d5be9186d16f6b06a"}
Jul 09 18:08:25.199981 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:25.199934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:08:25.220773 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:25.220705    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" podStartSLOduration=17.220690282 podStartE2EDuration="17.220690282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:25.220074787 +0000 UTC m=+236.321601839" watchObservedRunningTime="2026-07-09 18:08:25.220690282 +0000 UTC m=+236.322217308"
Jul 09 18:08:29.227819 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.227787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:29.227819 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.227822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:29.228377 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.228359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44a23042-fd7e-4bce-8ba9-d760919382ec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:29.230042 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.230023    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44a23042-fd7e-4bce-8ba9-d760919382ec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nqprt\" (UID: \"44a23042-fd7e-4bce-8ba9-d760919382ec\") " pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:29.291356 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.291331    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lw2dl\""
Jul 09 18:08:29.299357 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.299333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nqprt"
Jul 09 18:08:29.425170 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:29.425140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nqprt"]
Jul 09 18:08:29.429044 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:29.429017    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a23042_fd7e_4bce_8ba9_d760919382ec.slice/crio-80e9091b98534a833c6458da3650cae3dff5c7df12e3aeb5ffabf86fa1ce278a WatchSource:0}: Error finding container 80e9091b98534a833c6458da3650cae3dff5c7df12e3aeb5ffabf86fa1ce278a: Status 404 returned error can't find the container with id 80e9091b98534a833c6458da3650cae3dff5c7df12e3aeb5ffabf86fa1ce278a
Jul 09 18:08:30.217637 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:30.217594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqprt" event={"ID":"44a23042-fd7e-4bce-8ba9-d760919382ec","Type":"ContainerStarted","Data":"80e9091b98534a833c6458da3650cae3dff5c7df12e3aeb5ffabf86fa1ce278a"}
Jul 09 18:08:33.449789 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:33.449759    2561 scope.go:117] "RemoveContainer" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"
Jul 09 18:08:34.232000 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.231924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:08:34.232320 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.232303    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/1.log"
Jul 09 18:08:34.232413 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.232337    2561 generic.go:356] "Generic (PLEG): container finished" podID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e" containerID="913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9" exitCode=255
Jul 09 18:08:34.232452 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.232413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jxn6g" event={"ID":"00a0f1dd-da67-4730-9820-8fc0ec8ac69e","Type":"ContainerDied","Data":"913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9"}
Jul 09 18:08:34.232499 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.232469    2561 scope.go:117] "RemoveContainer" containerID="cd9b80cb9e2bc8e3766f05cb5c54df4d92831489af2679810b7c25708712d22f"
Jul 09 18:08:34.232825 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.232801    2561 scope.go:117] "RemoveContainer" containerID="913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9"
Jul 09 18:08:34.233030 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:34.232961    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:34.234129 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.234107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqprt" event={"ID":"44a23042-fd7e-4bce-8ba9-d760919382ec","Type":"ContainerStarted","Data":"f1a518b91113e028f14d38b4119f0290280f6ca450672ede338644862090c86e"}
Jul 09 18:08:34.234214 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:34.234149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqprt" event={"ID":"44a23042-fd7e-4bce-8ba9-d760919382ec","Type":"ContainerStarted","Data":"a33c421bf3970c0fb60393c08c3f287d01306420f44180409c3822bea470cf66"}
Jul 09 18:08:35.244546 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.244510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:08:35.800733 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.800686    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:08:35.862692 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.862659    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cc8ccb7bd-kdptw"]
Jul 09 18:08:35.870472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.870444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.887045 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.887006    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc8ccb7bd-kdptw"]
Jul 09 18:08:35.982443 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-installation-pull-secrets\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982443 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7831265e-7320-419f-ac58-048fa4d5b8b6-ca-trust-extracted\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-image-registry-private-configuration\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-certificates\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-bound-sa-token\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4q9\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-kube-api-access-tx4q9\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982948 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-tls\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:35.982948 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:35.982790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-trusted-ca\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-certificates\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-bound-sa-token\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4q9\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-kube-api-access-tx4q9\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084517 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-tls\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084517 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-trusted-ca\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084517 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-installation-pull-secrets\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084517 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7831265e-7320-419f-ac58-048fa4d5b8b6-ca-trust-extracted\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.084517 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.084410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-image-registry-private-configuration\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.085110 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.085017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7831265e-7320-419f-ac58-048fa4d5b8b6-ca-trust-extracted\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.085110 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.085066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-certificates\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.085495 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.085472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7831265e-7320-419f-ac58-048fa4d5b8b6-trusted-ca\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.087158 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.087134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-image-registry-private-configuration\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.087278 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.087166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-registry-tls\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.087433 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.087406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7831265e-7320-419f-ac58-048fa4d5b8b6-installation-pull-secrets\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.092081 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.092056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-bound-sa-token\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.095540 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.095518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4q9\" (UniqueName: \"kubernetes.io/projected/7831265e-7320-419f-ac58-048fa4d5b8b6-kube-api-access-tx4q9\") pod \"image-registry-cc8ccb7bd-kdptw\" (UID: \"7831265e-7320-419f-ac58-048fa4d5b8b6\") " pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.180940 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.180902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:36.571058 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:36.571030    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cc8ccb7bd-kdptw"]
Jul 09 18:08:36.573868 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:36.573840    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7831265e_7320_419f_ac58_048fa4d5b8b6.slice/crio-d90bc0a4f3c57f148f42584699e55af8f2e69dd93ffbd58781fd06358ed08c58 WatchSource:0}: Error finding container d90bc0a4f3c57f148f42584699e55af8f2e69dd93ffbd58781fd06358ed08c58: Status 404 returned error can't find the container with id d90bc0a4f3c57f148f42584699e55af8f2e69dd93ffbd58781fd06358ed08c58
Jul 09 18:08:37.251547 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.251509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw" event={"ID":"7831265e-7320-419f-ac58-048fa4d5b8b6","Type":"ContainerStarted","Data":"c8d1a44ffe9b2f7fba4f1e7024896e94d98a3e726a0e7e19d1ebec5fb97ed502"}
Jul 09 18:08:37.251760 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.251562    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw" event={"ID":"7831265e-7320-419f-ac58-048fa4d5b8b6","Type":"ContainerStarted","Data":"d90bc0a4f3c57f148f42584699e55af8f2e69dd93ffbd58781fd06358ed08c58"}
Jul 09 18:08:37.251760 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.251633    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:37.253367 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.253347    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nqprt" event={"ID":"44a23042-fd7e-4bce-8ba9-d760919382ec","Type":"ContainerStarted","Data":"a933b843b9a045c34acd901ef63cdf64842505a777a6705ebcbc6cc02e2b0546"}
Jul 09 18:08:37.273142 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.273095    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw" podStartSLOduration=2.273081259 podStartE2EDuration="2.273081259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:37.272252289 +0000 UTC m=+248.373779320" watchObservedRunningTime="2026-07-09 18:08:37.273081259 +0000 UTC m=+248.374608265"
Jul 09 18:08:37.294966 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:37.294917    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nqprt" podStartSLOduration=17.330485364 podStartE2EDuration="24.294902552s" totalImagesPullingTime="6.964417188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:08:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:37.294049502 +0000 UTC m=+248.395576551" watchObservedRunningTime="2026-07-09 18:08:37.294902552 +0000 UTC m=+248.396429590"
Jul 09 18:08:39.148456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:39.148418    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:39.148901 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:39.148474    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:08:39.148901 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:39.148760    2561 scope.go:117] "RemoveContainer" containerID="913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9"
Jul 09 18:08:39.148901 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:39.148893    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:40.517776 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:40.517733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:40.518300 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:40.518282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac28916c-868d-41f0-bfbf-b1ec473d8903-service-ca-bundle\") pod \"router-default-6f565474d4-kwrxx\" (UID: \"ac28916c-868d-41f0-bfbf-b1ec473d8903\") " pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:40.765992 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:40.765960    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pg6hn\""
Jul 09 18:08:40.774771 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:40.774707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:40.900940 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:40.900905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f565474d4-kwrxx"]
Jul 09 18:08:40.903461 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:40.903429    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac28916c_868d_41f0_bfbf_b1ec473d8903.slice/crio-6cae15078af57d4bd911563a588d36cf2fa159bc5389a0010b2e4c2c223c4203 WatchSource:0}: Error finding container 6cae15078af57d4bd911563a588d36cf2fa159bc5389a0010b2e4c2c223c4203: Status 404 returned error can't find the container with id 6cae15078af57d4bd911563a588d36cf2fa159bc5389a0010b2e4c2c223c4203
Jul 09 18:08:41.223017 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.222973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:08:41.225230 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.225212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5f59614-c402-4162-87a3-ab09262ca067-metrics-certs\") pod \"network-metrics-daemon-blwkj\" (UID: \"c5f59614-c402-4162-87a3-ab09262ca067\") " pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:08:41.266900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.266872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f565474d4-kwrxx" event={"ID":"ac28916c-868d-41f0-bfbf-b1ec473d8903","Type":"ContainerStarted","Data":"9393d63880988c7bcf6fde576d669e8afff9a0254efeac40d2019cdaa42d0120"}
Jul 09 18:08:41.267057 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.266918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f565474d4-kwrxx" event={"ID":"ac28916c-868d-41f0-bfbf-b1ec473d8903","Type":"ContainerStarted","Data":"6cae15078af57d4bd911563a588d36cf2fa159bc5389a0010b2e4c2c223c4203"}
Jul 09 18:08:41.284636 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.284583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f565474d4-kwrxx" podStartSLOduration=33.284569224 podStartE2EDuration="33.284569224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:41.283473718 +0000 UTC m=+252.385000744" watchObservedRunningTime="2026-07-09 18:08:41.284569224 +0000 UTC m=+252.386096251"
Jul 09 18:08:41.352902 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.352868    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7k8ns\""
Jul 09 18:08:41.361160 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.361141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blwkj"
Jul 09 18:08:41.500552 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.500392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blwkj"]
Jul 09 18:08:41.502815 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:41.502786    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f59614_c402_4162_87a3_ab09262ca067.slice/crio-1466968f08a3a860bdfda7dfd3f0931613d4524c53e5afd684445fee838dde21 WatchSource:0}: Error finding container 1466968f08a3a860bdfda7dfd3f0931613d4524c53e5afd684445fee838dde21: Status 404 returned error can't find the container with id 1466968f08a3a860bdfda7dfd3f0931613d4524c53e5afd684445fee838dde21
Jul 09 18:08:41.775618 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.775543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:41.777740 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:41.777703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:42.272503 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:42.272467    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blwkj" event={"ID":"c5f59614-c402-4162-87a3-ab09262ca067","Type":"ContainerStarted","Data":"1466968f08a3a860bdfda7dfd3f0931613d4524c53e5afd684445fee838dde21"}
Jul 09 18:08:42.272763 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:42.272745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:42.274289 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:42.274252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f565474d4-kwrxx"
Jul 09 18:08:43.277691 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:43.277661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blwkj" event={"ID":"c5f59614-c402-4162-87a3-ab09262ca067","Type":"ContainerStarted","Data":"ba6d5b4a42b3467dc91974acca1b26b978abd36ea701abfb6251874b162584f8"}
Jul 09 18:08:43.278080 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:43.277713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blwkj" event={"ID":"c5f59614-c402-4162-87a3-ab09262ca067","Type":"ContainerStarted","Data":"404924dfc8d6ac31d13c899743ebfc1007ad8cd6e1633b515bc9c5f84567dedd"}
Jul 09 18:08:43.303176 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:43.303119    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-blwkj" podStartSLOduration=253.363061594 podStartE2EDuration="4m14.303097021s" totalImagesPullingTime="940.035427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:43.302322235 +0000 UTC m=+254.403849264" watchObservedRunningTime="2026-07-09 18:08:43.303097021 +0000 UTC m=+254.404624049"
Jul 09 18:08:45.805903 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:45.805868    2561 patch_prober.go:28] interesting pod/image-registry-7c978d5698-jzqrs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:08:45.806346 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:45.805920    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:08:50.936139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.936067    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xcswq"]
Jul 09 18:08:50.938340 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.938323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.941383 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941349    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:08:50.941573 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:08:50.941573 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941562    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:08:50.941695 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941598    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xkn2w\""
Jul 09 18:08:50.941940 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941925    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:08:50.941992 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.941925    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:08:50.942378 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.942362    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:08:50.996804 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-tls\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-metrics-client-ca\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-sys\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-kube-api-access-z25rj\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.996982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997213 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.997009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-wtmp\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997213 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.997057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-root\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:50.997213 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:50.997080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-textfile\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.097825 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-root\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.097825 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-textfile\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-tls\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-root\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-metrics-client-ca\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.097999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-sys\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-kube-api-access-z25rj\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098082 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-wtmp\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-sys\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-wtmp\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-textfile\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098620 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.098812 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.098796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-metrics-client-ca\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.100267 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.100250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-tls\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.100357 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.100318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.105311 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.105291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/5a7f5f9e-d482-483f-b60f-03a8f42da0ff-kube-api-access-z25rj\") pod \"node-exporter-xcswq\" (UID: \"5a7f5f9e-d482-483f-b60f-03a8f42da0ff\") " pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.247699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.247608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcswq"
Jul 09 18:08:51.256746 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:51.256703    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7f5f9e_d482_483f_b60f_03a8f42da0ff.slice/crio-58a2729cf710c1bc52e9b81ad09d51c7d09fdb6026fabf542a098af95893c338 WatchSource:0}: Error finding container 58a2729cf710c1bc52e9b81ad09d51c7d09fdb6026fabf542a098af95893c338: Status 404 returned error can't find the container with id 58a2729cf710c1bc52e9b81ad09d51c7d09fdb6026fabf542a098af95893c338
Jul 09 18:08:51.303553 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:51.303523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcswq" event={"ID":"5a7f5f9e-d482-483f-b60f-03a8f42da0ff","Type":"ContainerStarted","Data":"58a2729cf710c1bc52e9b81ad09d51c7d09fdb6026fabf542a098af95893c338"}
Jul 09 18:08:53.313502 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:53.313470    2561 generic.go:356] "Generic (PLEG): container finished" podID="5a7f5f9e-d482-483f-b60f-03a8f42da0ff" containerID="6b34e2e30070c930afd052f87b73da42746c90351c1a02f6169a65bf25b7a999" exitCode=0
Jul 09 18:08:53.313502 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:53.313505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcswq" event={"ID":"5a7f5f9e-d482-483f-b60f-03a8f42da0ff","Type":"ContainerDied","Data":"6b34e2e30070c930afd052f87b73da42746c90351c1a02f6169a65bf25b7a999"}
Jul 09 18:08:53.449262 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:53.449232    2561 scope.go:117] "RemoveContainer" containerID="913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9"
Jul 09 18:08:53.449464 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:53.449390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-jxn6g_openshift-console-operator(00a0f1dd-da67-4730-9820-8fc0ec8ac69e)\"" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podUID="00a0f1dd-da67-4730-9820-8fc0ec8ac69e"
Jul 09 18:08:54.318604 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:54.318572    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcswq" event={"ID":"5a7f5f9e-d482-483f-b60f-03a8f42da0ff","Type":"ContainerStarted","Data":"d9db2414b2368da5b24372db82a0098d9ed5c6a6789d0a9fd6c1c85fd4ffd1ba"}
Jul 09 18:08:54.318990 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:54.318627    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcswq" event={"ID":"5a7f5f9e-d482-483f-b60f-03a8f42da0ff","Type":"ContainerStarted","Data":"226f59bcf09fe89606daae7489b4695a9ec7819a1c573d41806ded009df761d4"}
Jul 09 18:08:54.337393 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:54.337347    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xcswq" podStartSLOduration=3.193083526 podStartE2EDuration="4.337331465s" totalImagesPullingTime="1.079862512s" totalInitContainerRuntime="64.385427ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:54.335943397 +0000 UTC m=+265.437470423" watchObservedRunningTime="2026-07-09 18:08:54.337331465 +0000 UTC m=+265.438858493"
Jul 09 18:08:55.620362 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.620325    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"]
Jul 09 18:08:55.622307 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.622291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:55.625065 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.625043    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vgbqj\""
Jul 09 18:08:55.625333 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.625319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 09 18:08:55.635460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.635434    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"]
Jul 09 18:08:55.736116 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.736076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:55.736291 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.736137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbj9l\" (UniqueName: \"kubernetes.io/projected/2d300504-58da-4c86-ab1d-47d7376f0abb-kube-api-access-tbj9l\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:55.805259 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.805227    2561 patch_prober.go:28] interesting pod/image-registry-7c978d5698-jzqrs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:08:55.805414 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.805274    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:08:55.837292 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.837261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:55.837397 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.837310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj9l\" (UniqueName: \"kubernetes.io/projected/2d300504-58da-4c86-ab1d-47d7376f0abb-kube-api-access-tbj9l\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:55.837434 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:55.837408    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 09 18:08:55.837491 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:08:55.837480    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert podName:2d300504-58da-4c86-ab1d-47d7376f0abb nodeName:}" failed. No retries permitted until 2026-07-09 18:08:56.337456424 +0000 UTC m=+267.438983428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-rbqb5" (UID: "2d300504-58da-4c86-ab1d-47d7376f0abb") : secret "monitoring-plugin-cert" not found
Jul 09 18:08:55.847627 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:55.847597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj9l\" (UniqueName: \"kubernetes.io/projected/2d300504-58da-4c86-ab1d-47d7376f0abb-kube-api-access-tbj9l\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:56.185781 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.185739    2561 patch_prober.go:28] interesting pod/image-registry-cc8ccb7bd-kdptw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:08:56.185980 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.185790    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw" podUID="7831265e-7320-419f-ac58-048fa4d5b8b6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:08:56.341874 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.341839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:56.344141 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.344114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d300504-58da-4c86-ab1d-47d7376f0abb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rbqb5\" (UID: \"2d300504-58da-4c86-ab1d-47d7376f0abb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:56.531146 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.531066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:56.654033 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:56.653915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"]
Jul 09 18:08:56.656514 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:56.656490    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d300504_58da_4c86_ab1d_47d7376f0abb.slice/crio-66537c474749503fb189a0ac3ca79f39281573992d6448a5a92fa3eb05eb10d9 WatchSource:0}: Error finding container 66537c474749503fb189a0ac3ca79f39281573992d6448a5a92fa3eb05eb10d9: Status 404 returned error can't find the container with id 66537c474749503fb189a0ac3ca79f39281573992d6448a5a92fa3eb05eb10d9
Jul 09 18:08:57.108704 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.108671    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:08:57.112033 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.112015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.115211 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.115183    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:08:57.115211 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.115172    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:08:57.116013 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.115992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:08:57.116376 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.116356    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:08:57.117013 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.116993    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:08:57.117124 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.117107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:08:57.118605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118586    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8e6e11jd2vim6\""
Jul 09 18:08:57.118770 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118738    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:08:57.118893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:08:57.118893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118818    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:08:57.118893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118744    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4rm8q\""
Jul 09 18:08:57.119394 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.118961    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:08:57.119394 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.119102    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:08:57.119491 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.119411    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:08:57.119633 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.119612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:08:57.120559 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.120541    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:08:57.123829 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.123811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:08:57.138228 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.138202    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:08:57.249019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.248982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249493 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4d2\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.249798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.249634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.329921 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.329885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5" event={"ID":"2d300504-58da-4c86-ab1d-47d7376f0abb","Type":"ContainerStarted","Data":"66537c474749503fb189a0ac3ca79f39281573992d6448a5a92fa3eb05eb10d9"}
Jul 09 18:08:57.350235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350538    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.350755 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4d2\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.350948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351495 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.351433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.351545 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.351516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.353653 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.353628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.353938 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.353918    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.354026 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.353925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.354105 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.354085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.354597 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.354472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.354597 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.354514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.355001 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.354697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.355491 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.355168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.355491 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.355289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.355710 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.355691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.356028 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.356005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.356108 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.356073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.356673 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.356644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.356931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.356911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.357090 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.357077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.358074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.358058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.362060 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.362011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4d2\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2\") pod \"prometheus-k8s-0\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.424144 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.424106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:08:57.567039 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:57.566942    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:08:57.572308 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:08:57.570589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ffd1c9_23ad_4720_a56d_e54d79b063a9.slice/crio-c8220b09996e5689e8b829f33a5741ceb24024997ccc9d2fe1e583bb070b5328 WatchSource:0}: Error finding container c8220b09996e5689e8b829f33a5741ceb24024997ccc9d2fe1e583bb070b5328: Status 404 returned error can't find the container with id c8220b09996e5689e8b829f33a5741ceb24024997ccc9d2fe1e583bb070b5328
Jul 09 18:08:58.262221 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:58.262193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cc8ccb7bd-kdptw"
Jul 09 18:08:58.335050 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:58.335005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"c8220b09996e5689e8b829f33a5741ceb24024997ccc9d2fe1e583bb070b5328"}
Jul 09 18:08:59.340005 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.339969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5" event={"ID":"2d300504-58da-4c86-ab1d-47d7376f0abb","Type":"ContainerStarted","Data":"929958b32fbc4398a344b16009492dfa6f071e05cc5fc4859c6b6148b06bd7ab"}
Jul 09 18:08:59.340440 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.340023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:59.341688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.341666    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="7cd4e1251ac1a9f7b015c3085cd0c8cc9ab4b89a5167b667f594ac05258e1eb9" exitCode=0
Jul 09 18:08:59.341830 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.341705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"7cd4e1251ac1a9f7b015c3085cd0c8cc9ab4b89a5167b667f594ac05258e1eb9"}
Jul 09 18:08:59.345476 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.345442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5"
Jul 09 18:08:59.354556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:08:59.354517    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rbqb5" podStartSLOduration=2.21761005 podStartE2EDuration="4.354502927s" totalImagesPullingTime="2.136892877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:08:59.353828006 +0000 UTC m=+270.455355033" watchObservedRunningTime="2026-07-09 18:08:59.354502927 +0000 UTC m=+270.456029985"
Jul 09 18:09:00.822430 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:00.822383    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry" containerID="cri-o://a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7" gracePeriod=30
Jul 09 18:09:01.077793 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.077710    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:09:01.185625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzfbm\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185786    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.185895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.185869    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") pod \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\" (UID: \"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9\") "
Jul 09 18:09:01.186323 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.186287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:09:01.186420 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.186380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:09:01.188646 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.188616    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:09:01.189487 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.189434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:09:01.189487 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.189451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:09:01.189631 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.189541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:09:01.189817 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.189793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm" (OuterVolumeSpecName: "kube-api-access-qzfbm") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "kube-api-access-qzfbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:09:01.196991 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.196968    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" (UID: "d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:09:01.286742 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286697    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-trusted-ca\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286742 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286740    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-image-registry-private-configuration\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286757    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286770    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzfbm\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-kube-api-access-qzfbm\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286779    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-bound-sa-token\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286788    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-registry-certificates\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286795    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-ca-trust-extracted\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.286945 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.286803    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9-installation-pull-secrets\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:09:01.350379 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.350298    2561 generic.go:356] "Generic (PLEG): container finished" podID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerID="a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7" exitCode=0
Jul 09 18:09:01.350379 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.350360    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs"
Jul 09 18:09:01.350586 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.350386    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" event={"ID":"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9","Type":"ContainerDied","Data":"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"}
Jul 09 18:09:01.350586 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.350474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c978d5698-jzqrs" event={"ID":"d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9","Type":"ContainerDied","Data":"b51ddac51d0c6e662685482192fed9d9cfb0af54b4b1f50d5be9186d16f6b06a"}
Jul 09 18:09:01.350586 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.350501    2561 scope.go:117] "RemoveContainer" containerID="a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"
Jul 09 18:09:01.359828 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.359808    2561 scope.go:117] "RemoveContainer" containerID="a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"
Jul 09 18:09:01.360220 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:09:01.360171    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7\": container with ID starting with a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7 not found: ID does not exist" containerID="a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"
Jul 09 18:09:01.360305 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.360227    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7"} err="failed to get container status \"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7\": rpc error: code = NotFound desc = could not find container \"a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7\": container with ID starting with a2afa1de86394371024e4e20a91b731701659c2a9e4ebb3060e5cb2da8ba02d7 not found: ID does not exist"
Jul 09 18:09:01.372780 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.372752    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:09:01.376453 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.376432    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c978d5698-jzqrs"]
Jul 09 18:09:01.454577 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:01.454543    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" path="/var/lib/kubelet/pods/d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9/volumes"
Jul 09 18:09:03.360794 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:03.360756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"70ba9d39838a1c4317947cf84be58e60f4356fe30fcad67da0da839851e4ac39"}
Jul 09 18:09:03.361181 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:03.360834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"5c251f53b5a271a143c8488ba2baf225f7f7d127ad410d22e8e464198380e57e"}
Jul 09 18:09:06.375596 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:06.375564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"1924e21adb697b330577ae68a54c6f5ed6a4ef5e8052159799210584873d6614"}
Jul 09 18:09:06.375934 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:06.375678    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"d0d42602609ee0ef543eba4b99877cc7839ed74bdfe0fb7d8d9dcac17ef6632b"}
Jul 09 18:09:07.381799 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:07.381761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"1bda5634a29e4710124cd71e17c6663da2175d20f263fcd30d29897657cb52b1"}
Jul 09 18:09:07.382363 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:07.381842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerStarted","Data":"b1ad458be6943f58b094c0a91e02a10c0dbaae38e469c29a5b50b23eb7b7a21c"}
Jul 09 18:09:07.419171 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:07.419106    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.444910523 podStartE2EDuration="10.41908222s" totalImagesPullingTime="7.951767831s" totalInitContainerRuntime="22.403866ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:07.418468133 +0000 UTC m=+278.519995203" watchObservedRunningTime="2026-07-09 18:09:07.41908222 +0000 UTC m=+278.520609248"
Jul 09 18:09:07.424404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:07.424375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:07.450114 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:07.450088    2561 scope.go:117] "RemoveContainer" containerID="913bdd9ca0e64b64eeb7f519d30ee8d9d94acadd6c5f55bb63d03a90f9b013e9"
Jul 09 18:09:08.387837 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:08.387804    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:09:08.388288 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:08.387923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jxn6g" event={"ID":"00a0f1dd-da67-4730-9820-8fc0ec8ac69e","Type":"ContainerStarted","Data":"8fa93ab7819375d4bad3dd8346fa5cc7f8176ccf0c4b474651ca5d41d07895a0"}
Jul 09 18:09:08.408008 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:08.407959    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-jxn6g" podStartSLOduration=58.173624903 podStartE2EDuration="1m0.407942195s" totalImagesPullingTime="2.234317292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:08.407053188 +0000 UTC m=+279.508580230" watchObservedRunningTime="2026-07-09 18:09:08.407942195 +0000 UTC m=+279.509469221"
Jul 09 18:09:10.964680 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:09:10.964636    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7mkq2" podUID="d56b0b17-5d17-4327-87d0-f5904cb3aa04"
Jul 09 18:09:11.396624 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:11.396600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:14.205716 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:14.205681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:14.208021 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:14.207995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d56b0b17-5d17-4327-87d0-f5904cb3aa04-metrics-tls\") pod \"dns-default-7mkq2\" (UID: \"d56b0b17-5d17-4327-87d0-f5904cb3aa04\") " pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:14.399619 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:14.399588    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jd75b\""
Jul 09 18:09:14.408439 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:14.408420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:14.541941 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:14.541920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mkq2"]
Jul 09 18:09:14.544202 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:09:14.544165    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b0b17_5d17_4327_87d0_f5904cb3aa04.slice/crio-97fa42bdbf1572406e8966ec364f380895896168a0050a6a28267a7e543b2fbf WatchSource:0}: Error finding container 97fa42bdbf1572406e8966ec364f380895896168a0050a6a28267a7e543b2fbf: Status 404 returned error can't find the container with id 97fa42bdbf1572406e8966ec364f380895896168a0050a6a28267a7e543b2fbf
Jul 09 18:09:15.412150 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:15.412112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mkq2" event={"ID":"d56b0b17-5d17-4327-87d0-f5904cb3aa04","Type":"ContainerStarted","Data":"97fa42bdbf1572406e8966ec364f380895896168a0050a6a28267a7e543b2fbf"}
Jul 09 18:09:16.416992 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:16.416955    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mkq2" event={"ID":"d56b0b17-5d17-4327-87d0-f5904cb3aa04","Type":"ContainerStarted","Data":"d0859196360cc1742acae6dc75a310d90b58fdbc78c1bd971206eb821a6311cf"}
Jul 09 18:09:16.417368 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:16.417053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mkq2" event={"ID":"d56b0b17-5d17-4327-87d0-f5904cb3aa04","Type":"ContainerStarted","Data":"78fd9224821aec824c1f13fb33f940b22139ee4d6a04ec49f489e20ed8256907"}
Jul 09 18:09:16.417368 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:16.417121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:16.434357 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:16.434300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7mkq2" podStartSLOduration=250.996917288 podStartE2EDuration="4m12.43427895s" totalImagesPullingTime="1.437361662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:16.433578561 +0000 UTC m=+287.535105588" watchObservedRunningTime="2026-07-09 18:09:16.43427895 +0000 UTC m=+287.535805976"
Jul 09 18:09:18.389200 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:18.389160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:09:18.392992 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:18.392973    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-jxn6g"
Jul 09 18:09:22.438825 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:22.438790    2561 generic.go:356] "Generic (PLEG): container finished" podID="fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9" containerID="36dd204a6a8c4b3c4caa9f3ff0d19ac4e2cf03eefef0624b30f34afe7a4cdcda" exitCode=0
Jul 09 18:09:22.439194 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:22.438864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d" event={"ID":"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9","Type":"ContainerDied","Data":"36dd204a6a8c4b3c4caa9f3ff0d19ac4e2cf03eefef0624b30f34afe7a4cdcda"}
Jul 09 18:09:22.439194 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:22.439156    2561 scope.go:117] "RemoveContainer" containerID="36dd204a6a8c4b3c4caa9f3ff0d19ac4e2cf03eefef0624b30f34afe7a4cdcda"
Jul 09 18:09:23.445621 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:23.445589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-5pm5d" event={"ID":"fdb263d2-3721-4f5c-bbda-fb44c9d4b1e9","Type":"ContainerStarted","Data":"cac9fd7a5f13bb1a2264c473b57b3c0c4b4cd0719a3894f6bd0ce252757cfd90"}
Jul 09 18:09:26.423426 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:26.423394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7mkq2"
Jul 09 18:09:29.285069 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:29.285047    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:09:29.285069 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:29.285047    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:09:29.290188 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:29.290171    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:09:29.290384 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:29.290369    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:09:29.297740 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:29.297696    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:09:37.473274 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:37.473245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-rbqb5_2d300504-58da-4c86-ab1d-47d7376f0abb/monitoring-plugin/0.log"
Jul 09 18:09:38.875166 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:38.875131    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/init-textfile/0.log"
Jul 09 18:09:39.075231 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:39.075206    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/node-exporter/0.log"
Jul 09 18:09:39.274155 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:39.274090    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/kube-rbac-proxy/0.log"
Jul 09 18:09:40.073547 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:40.073509    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/init-config-reloader/0.log"
Jul 09 18:09:40.275501 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:40.275456    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/prometheus/0.log"
Jul 09 18:09:40.474199 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:40.474112    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/config-reloader/0.log"
Jul 09 18:09:40.673784 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:40.673757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/thanos-sidecar/0.log"
Jul 09 18:09:40.872596 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:40.872574    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/kube-rbac-proxy-web/0.log"
Jul 09 18:09:41.072679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:41.072653    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/kube-rbac-proxy/0.log"
Jul 09 18:09:41.272951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:41.272879    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a2ffd1c9-23ad-4720-a56d-e54d79b063a9/kube-rbac-proxy-thanos/0.log"
Jul 09 18:09:44.073097 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:44.073071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:09:44.274696 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:44.274670    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/3.log"
Jul 09 18:09:45.072829 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:45.072802    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f565474d4-kwrxx_ac28916c-868d-41f0-bfbf-b1ec473d8903/router/0.log"
Jul 09 18:09:45.272576 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:45.272549    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5rmsk_648fd878-2053-4126-b5b7-6bad43fe0141/serve-healthcheck-canary/0.log"
Jul 09 18:09:57.425262 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:57.425212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:57.440855 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:57.440828    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:09:57.585369 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:09:57.585341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:15.488961 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.488925    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489239    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="prometheus" containerID="cri-o://5c251f53b5a271a143c8488ba2baf225f7f7d127ad410d22e8e464198380e57e" gracePeriod=600
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489272    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="thanos-sidecar" containerID="cri-o://d0d42602609ee0ef543eba4b99877cc7839ed74bdfe0fb7d8d9dcac17ef6632b" gracePeriod=600
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489303    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-web" containerID="cri-o://1924e21adb697b330577ae68a54c6f5ed6a4ef5e8052159799210584873d6614" gracePeriod=600
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489276    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1bda5634a29e4710124cd71e17c6663da2175d20f263fcd30d29897657cb52b1" gracePeriod=600
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489304    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy" containerID="cri-o://b1ad458be6943f58b094c0a91e02a10c0dbaae38e469c29a5b50b23eb7b7a21c" gracePeriod=600
Jul 09 18:10:15.489563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.489278    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="config-reloader" containerID="cri-o://70ba9d39838a1c4317947cf84be58e60f4356fe30fcad67da0da839851e4ac39" gracePeriod=600
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638111    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="1bda5634a29e4710124cd71e17c6663da2175d20f263fcd30d29897657cb52b1" exitCode=0
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638134    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="b1ad458be6943f58b094c0a91e02a10c0dbaae38e469c29a5b50b23eb7b7a21c" exitCode=0
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638141    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="1924e21adb697b330577ae68a54c6f5ed6a4ef5e8052159799210584873d6614" exitCode=0
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638147    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="d0d42602609ee0ef543eba4b99877cc7839ed74bdfe0fb7d8d9dcac17ef6632b" exitCode=0
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638152    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="70ba9d39838a1c4317947cf84be58e60f4356fe30fcad67da0da839851e4ac39" exitCode=0
Jul 09 18:10:15.638139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638157    2561 generic.go:356] "Generic (PLEG): container finished" podID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerID="5c251f53b5a271a143c8488ba2baf225f7f7d127ad410d22e8e464198380e57e" exitCode=0
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"1bda5634a29e4710124cd71e17c6663da2175d20f263fcd30d29897657cb52b1"}
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"b1ad458be6943f58b094c0a91e02a10c0dbaae38e469c29a5b50b23eb7b7a21c"}
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"1924e21adb697b330577ae68a54c6f5ed6a4ef5e8052159799210584873d6614"}
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"d0d42602609ee0ef543eba4b99877cc7839ed74bdfe0fb7d8d9dcac17ef6632b"}
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"70ba9d39838a1c4317947cf84be58e60f4356fe30fcad67da0da839851e4ac39"}
Jul 09 18:10:15.638437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.638355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"5c251f53b5a271a143c8488ba2baf225f7f7d127ad410d22e8e464198380e57e"}
Jul 09 18:10:15.733943 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.733912    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:15.841016 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.840983    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841016 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841135    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841264 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841312    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c4d2\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841407    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841410    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841431    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841468    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841503    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.841566 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841531    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.842078 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.842078 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.842078 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy\") pod \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\" (UID: \"a2ffd1c9-23ad-4720-a56d-e54d79b063a9\") "
Jul 09 18:10:15.842078 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841891    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.842420 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.842623 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.843292 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841565    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.843475 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.841871    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.844067 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.843562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.844195 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.844080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out" (OuterVolumeSpecName: "config-out") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:10:15.844462 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.844437    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:15.844563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.844543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2" (OuterVolumeSpecName: "kube-api-access-9c4d2") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "kube-api-access-9c4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:15.844905 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.844859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.845001 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.844959    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:10:15.845625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.845536    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.845625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.845595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.845799 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.845619    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.845799 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.845741    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.845914 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.845838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.846059 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.846036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:15.846059 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.846041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config" (OuterVolumeSpecName: "config") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.846818 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.846798    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.855841 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.855814    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config" (OuterVolumeSpecName: "web-config") pod "a2ffd1c9-23ad-4720-a56d-e54d79b063a9" (UID: "a2ffd1c9-23ad-4720-a56d-e54d79b063a9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:15.942261 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942218    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942261 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942256    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-db\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942261 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942268    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-web-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942279    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-kube-rbac-proxy\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942289    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942298    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config-out\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942306    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942315    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-configmap-metrics-client-ca\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942323    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942331    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-metrics-client-certs\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942339    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942349    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-tls-assets\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942358    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942367    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-grpc-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942374    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4d2\" (UniqueName: \"kubernetes.io/projected/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-kube-api-access-9c4d2\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942382    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942389    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942397    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:15.942480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:15.942405    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a2ffd1c9-23ad-4720-a56d-e54d79b063a9-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:10:16.645156 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.645118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a2ffd1c9-23ad-4720-a56d-e54d79b063a9","Type":"ContainerDied","Data":"c8220b09996e5689e8b829f33a5741ceb24024997ccc9d2fe1e583bb070b5328"}
Jul 09 18:10:16.645523 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.645203    2561 scope.go:117] "RemoveContainer" containerID="1bda5634a29e4710124cd71e17c6663da2175d20f263fcd30d29897657cb52b1"
Jul 09 18:10:16.645523 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.645133    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.654852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.654619    2561 scope.go:117] "RemoveContainer" containerID="b1ad458be6943f58b094c0a91e02a10c0dbaae38e469c29a5b50b23eb7b7a21c"
Jul 09 18:10:16.662195 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.662176    2561 scope.go:117] "RemoveContainer" containerID="1924e21adb697b330577ae68a54c6f5ed6a4ef5e8052159799210584873d6614"
Jul 09 18:10:16.668591 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.668574    2561 scope.go:117] "RemoveContainer" containerID="d0d42602609ee0ef543eba4b99877cc7839ed74bdfe0fb7d8d9dcac17ef6632b"
Jul 09 18:10:16.674786 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.674770    2561 scope.go:117] "RemoveContainer" containerID="70ba9d39838a1c4317947cf84be58e60f4356fe30fcad67da0da839851e4ac39"
Jul 09 18:10:16.681027 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.681013    2561 scope.go:117] "RemoveContainer" containerID="5c251f53b5a271a143c8488ba2baf225f7f7d127ad410d22e8e464198380e57e"
Jul 09 18:10:16.687769 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.687751    2561 scope.go:117] "RemoveContainer" containerID="7cd4e1251ac1a9f7b015c3085cd0c8cc9ab4b89a5167b667f594ac05258e1eb9"
Jul 09 18:10:16.690435 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.690415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:16.696581 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.696560    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:16.768850 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.768761    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:16.769081 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769064    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769084    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769099    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-thanos"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769108    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-thanos"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769122    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="prometheus"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="prometheus"
Jul 09 18:10:16.769145 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769140    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-web"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769148    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-web"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769173    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="config-reloader"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769182    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="config-reloader"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769213    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="init-config-reloader"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769220    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="init-config-reloader"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769229    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="thanos-sidecar"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769238    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="thanos-sidecar"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769308    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-web"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769321    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9bc18ba-74d1-4392-ad9e-ab3f4c659cc9" containerName="registry"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769331    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769341    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="prometheus"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769349    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="thanos-sidecar"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769357    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="kube-rbac-proxy-thanos"
Jul 09 18:10:16.769428 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.769366    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" containerName="config-reloader"
Jul 09 18:10:16.774574 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.774553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.778370 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.778350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 09 18:10:16.778370 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.778363    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 09 18:10:16.778897 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.778875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 09 18:10:16.779026 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 18:10:16.779139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4rm8q\""
Jul 09 18:10:16.779187 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779157    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 09 18:10:16.779187 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779150    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 09 18:10:16.779415 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 09 18:10:16.779501 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 09 18:10:16.779556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.779513    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 09 18:10:16.780867 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.780850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 09 18:10:16.781176 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.781155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 09 18:10:16.781601 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.781586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 09 18:10:16.781687 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.781634    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 09 18:10:16.782978 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.782653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8e6e11jd2vim6\""
Jul 09 18:10:16.787551 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.785766    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 09 18:10:16.793347 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.793328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 09 18:10:16.796135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.796110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:16.849836 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.849836 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.849992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tbp\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-kube-api-access-64tbp\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.850374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.850304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951596 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951596 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.951879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.951823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.952218 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tbp\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-kube-api-access-64tbp\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953051 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.953025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953810 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.953622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953810 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.952471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.953910 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.953891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.955006 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.954980    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.955214 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.955166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-web-config\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.956426 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.956407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.956935 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.956635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.956935 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.956878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f7139b-924f-43fe-ac3b-d696cde592d9-config-out\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.956998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957197 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957335 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32f7139b-924f-43fe-ac3b-d696cde592d9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957420 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957420 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957372    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957518 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.957586 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.957570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.958445 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.958430    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32f7139b-924f-43fe-ac3b-d696cde592d9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:16.961128 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:16.961107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tbp\" (UniqueName: \"kubernetes.io/projected/32f7139b-924f-43fe-ac3b-d696cde592d9-kube-api-access-64tbp\") pod \"prometheus-k8s-0\" (UID: \"32f7139b-924f-43fe-ac3b-d696cde592d9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:17.086359 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.086322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:10:17.229541 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.229481    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 09 18:10:17.231622 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:10:17.231599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f7139b_924f_43fe_ac3b_d696cde592d9.slice/crio-cbc4376d55ab7815803b384ce909610e861866e4c8ab6ca2377594547ed187cb WatchSource:0}: Error finding container cbc4376d55ab7815803b384ce909610e861866e4c8ab6ca2377594547ed187cb: Status 404 returned error can't find the container with id cbc4376d55ab7815803b384ce909610e861866e4c8ab6ca2377594547ed187cb
Jul 09 18:10:17.454282 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.454201    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ffd1c9-23ad-4720-a56d-e54d79b063a9" path="/var/lib/kubelet/pods/a2ffd1c9-23ad-4720-a56d-e54d79b063a9/volumes"
Jul 09 18:10:17.650944 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.650911    2561 generic.go:356] "Generic (PLEG): container finished" podID="32f7139b-924f-43fe-ac3b-d696cde592d9" containerID="91ea9a3f488ee5a0ab58a789433c21bdad07527bc440458433aab5760bc72e40" exitCode=0
Jul 09 18:10:17.651334 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.650993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerDied","Data":"91ea9a3f488ee5a0ab58a789433c21bdad07527bc440458433aab5760bc72e40"}
Jul 09 18:10:17.651334 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:17.651076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"cbc4376d55ab7815803b384ce909610e861866e4c8ab6ca2377594547ed187cb"}
Jul 09 18:10:18.658931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"7aeaf2dd9de9fb4bc23e11725bff864674aa6cbc916398135f93a3093b49b8a8"}
Jul 09 18:10:18.658931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"52887f3c6a64653d679d0fbcf9ee3e5c6d626579e03102f454f061f9b030982a"}
Jul 09 18:10:18.658931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"4aee020e2d1d4e3cd3bbb759dae5f9df79e27cb49645ff5de3b5a44a782ff582"}
Jul 09 18:10:18.658931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"530da10241a369c332f918366178e397608d0522972b622d349c119dfb63bea6"}
Jul 09 18:10:18.659404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"3f0a9847009fd1ea6aaa5b2c7f5997bbeb55bf48fb871485e527a934da34ba21"}
Jul 09 18:10:18.659404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.658978    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32f7139b-924f-43fe-ac3b-d696cde592d9","Type":"ContainerStarted","Data":"1805a8274bfc681a567949e952dd9b443426f07c64862de013c23e0b3ed3b0b5"}
Jul 09 18:10:18.687315 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:18.687241    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.669081712 podStartE2EDuration="2.687225847s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.144135ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:18.686939994 +0000 UTC m=+349.788467033" watchObservedRunningTime="2026-07-09 18:10:18.687225847 +0000 UTC m=+349.788752874"
Jul 09 18:10:22.086713 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:10:22.086627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:11:17.087494 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:11:17.087455    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:11:17.127601 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:11:17.127575    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:11:17.881353 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:11:17.881329    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 09 18:14:29.307810 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:14:29.307782    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:14:29.309309 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:14:29.309287    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:14:29.312210 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:14:29.312192    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:14:29.313910 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:14:29.313883    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:15:56.905936 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.905899    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:15:56.908975 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.908954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:56.911702 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.911673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6s7vj\""
Jul 09 18:15:56.911826 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.911801    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 18:15:56.912076 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.912053    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db4c0-kube-rbac-proxy-sar-config\""
Jul 09 18:15:56.912866 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.912847    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 18:15:56.913440 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.912872    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-db4c0-predictor-serving-cert\""
Jul 09 18:15:56.919187 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:56.919164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:15:57.052438 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.052400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqqqz\" (UniqueName: \"kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.052438 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.052441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.052663 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.052471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.153306 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.153272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqqqz\" (UniqueName: \"kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.153306 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.153307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.153515 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.153434    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.154111 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.154094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.155801 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.155784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.160416 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.160347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqqqz\" (UniqueName: \"kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz\") pod \"error-404-isvc-db4c0-predictor-8564b78944-z9xvd\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") " pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.224991 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.224950    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:15:57.351717 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.351694    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:15:57.354274 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:15:57.354242    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779a3756_a8ca_4057_94f0_6a217e8cb942.slice/crio-0554b8e1ceb41f35417067914f78654a988abae3e3740be54ee6457b99a727d7 WatchSource:0}: Error finding container 0554b8e1ceb41f35417067914f78654a988abae3e3740be54ee6457b99a727d7: Status 404 returned error can't find the container with id 0554b8e1ceb41f35417067914f78654a988abae3e3740be54ee6457b99a727d7
Jul 09 18:15:57.356135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.356120    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:15:57.839434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:15:57.839398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerStarted","Data":"0554b8e1ceb41f35417067914f78654a988abae3e3740be54ee6457b99a727d7"}
Jul 09 18:16:10.897484 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:10.897450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerStarted","Data":"3c3301262469292cfab50c6d7ee2cb291959fc98a4072ba0507f6e6ee7c9c659"}
Jul 09 18:16:14.914840 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:14.914801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerStarted","Data":"0f43aa66d826c2716c150579d1a50d1c97460e52cbdffd12352942d0bdea27d7"}
Jul 09 18:16:14.915319 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:14.914877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:16:14.915319 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:14.914920    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:16:14.916230 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:14.916205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:16:14.933057 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:14.933006    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podStartSLOduration=2.361362799 podStartE2EDuration="18.932992663s" totalImagesPullingTime="16.571629864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:15:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:16:14.931539431 +0000 UTC m=+706.033066475" watchObservedRunningTime="2026-07-09 18:16:14.932992663 +0000 UTC m=+706.034519689"
Jul 09 18:16:15.919462 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:15.919422    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:16:20.924093 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:20.924018    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:16:20.924591 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:20.924565    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:16:30.924772 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:30.924713    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:16:40.925468 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:40.925426    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:16:50.924889 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:16:50.924843    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 09 18:17:00.924920 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:00.924883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:17:40.921556 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.921523    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:17:40.922151 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.921855    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container" containerID="cri-o://3c3301262469292cfab50c6d7ee2cb291959fc98a4072ba0507f6e6ee7c9c659" gracePeriod=30
Jul 09 18:17:40.922151 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.921850    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kube-rbac-proxy" containerID="cri-o://0f43aa66d826c2716c150579d1a50d1c97460e52cbdffd12352942d0bdea27d7" gracePeriod=30
Jul 09 18:17:40.925194 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.925172    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:17:40.927893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.927850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:40.930163 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.930144    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2a7bf-kube-rbac-proxy-sar-config\""
Jul 09 18:17:40.930248 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.930191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2a7bf-predictor-serving-cert\""
Jul 09 18:17:40.940279 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:40.940255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:17:41.114782 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.114741    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nwg\" (UniqueName: \"kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.114963 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.114821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.114963 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.114873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.216178 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.216091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.216178 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.216143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.216178 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.216180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nwg\" (UniqueName: \"kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.216955 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.216932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.218472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.218449    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.223828 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.223801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nwg\" (UniqueName: \"kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg\") pod \"error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") " pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.234243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.234218    2561 generic.go:356] "Generic (PLEG): container finished" podID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerID="0f43aa66d826c2716c150579d1a50d1c97460e52cbdffd12352942d0bdea27d7" exitCode=2
Jul 09 18:17:41.234337 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.234276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerDied","Data":"0f43aa66d826c2716c150579d1a50d1c97460e52cbdffd12352942d0bdea27d7"}
Jul 09 18:17:41.243432 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.243418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:41.370392 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:41.370364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:17:41.372842 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:17:41.372812    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8aa7b4_3c06_4718_bf0c_24bf4ed449a8.slice/crio-8a712afab6e1c77abe9464a6ccf0c9a774d38b5cc876fa0026ff550e19955ce5 WatchSource:0}: Error finding container 8a712afab6e1c77abe9464a6ccf0c9a774d38b5cc876fa0026ff550e19955ce5: Status 404 returned error can't find the container with id 8a712afab6e1c77abe9464a6ccf0c9a774d38b5cc876fa0026ff550e19955ce5
Jul 09 18:17:42.239444 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:42.239407    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerStarted","Data":"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"}
Jul 09 18:17:42.239842 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:42.239466    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:42.239842 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:42.239487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerStarted","Data":"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"}
Jul 09 18:17:42.239842 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:42.239506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerStarted","Data":"8a712afab6e1c77abe9464a6ccf0c9a774d38b5cc876fa0026ff550e19955ce5"}
Jul 09 18:17:42.256217 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:42.256162    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podStartSLOduration=2.256145768 podStartE2EDuration="2.256145768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:42.254989679 +0000 UTC m=+793.356516708" watchObservedRunningTime="2026-07-09 18:17:42.256145768 +0000 UTC m=+793.357672795"
Jul 09 18:17:43.243665 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:43.243627    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:43.244892 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:43.244863    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:17:44.248801 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.248772    2561 generic.go:356] "Generic (PLEG): container finished" podID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerID="3c3301262469292cfab50c6d7ee2cb291959fc98a4072ba0507f6e6ee7c9c659" exitCode=0
Jul 09 18:17:44.249184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.248846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerDied","Data":"3c3301262469292cfab50c6d7ee2cb291959fc98a4072ba0507f6e6ee7c9c659"}
Jul 09 18:17:44.249184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.249157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:17:44.267289 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.267269    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:17:44.444424 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.444338    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls\") pod \"779a3756-a8ca-4057-94f0-6a217e8cb942\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") "
Jul 09 18:17:44.444424 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.444384    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqqqz\" (UniqueName: \"kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz\") pod \"779a3756-a8ca-4057-94f0-6a217e8cb942\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") "
Jul 09 18:17:44.444623 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.444441    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config\") pod \"779a3756-a8ca-4057-94f0-6a217e8cb942\" (UID: \"779a3756-a8ca-4057-94f0-6a217e8cb942\") "
Jul 09 18:17:44.444857 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.444833    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "779a3756-a8ca-4057-94f0-6a217e8cb942" (UID: "779a3756-a8ca-4057-94f0-6a217e8cb942"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:17:44.446612 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.446585    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz" (OuterVolumeSpecName: "kube-api-access-jqqqz") pod "779a3756-a8ca-4057-94f0-6a217e8cb942" (UID: "779a3756-a8ca-4057-94f0-6a217e8cb942"). InnerVolumeSpecName "kube-api-access-jqqqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:44.446740 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.446623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "779a3756-a8ca-4057-94f0-6a217e8cb942" (UID: "779a3756-a8ca-4057-94f0-6a217e8cb942"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:44.545139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.545113    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:17:44.545139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.545137    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/779a3756-a8ca-4057-94f0-6a217e8cb942-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:17:44.545139 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:44.545148    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqqqz\" (UniqueName: \"kubernetes.io/projected/779a3756-a8ca-4057-94f0-6a217e8cb942-kube-api-access-jqqqz\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:17:45.258554 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.258521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd" event={"ID":"779a3756-a8ca-4057-94f0-6a217e8cb942","Type":"ContainerDied","Data":"0554b8e1ceb41f35417067914f78654a988abae3e3740be54ee6457b99a727d7"}
Jul 09 18:17:45.258554 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.258546    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"
Jul 09 18:17:45.259006 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.258615    2561 scope.go:117] "RemoveContainer" containerID="0f43aa66d826c2716c150579d1a50d1c97460e52cbdffd12352942d0bdea27d7"
Jul 09 18:17:45.267060 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.267041    2561 scope.go:117] "RemoveContainer" containerID="3c3301262469292cfab50c6d7ee2cb291959fc98a4072ba0507f6e6ee7c9c659"
Jul 09 18:17:45.278209 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.278188    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:17:45.281717 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.281693    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-db4c0-predictor-8564b78944-z9xvd"]
Jul 09 18:17:45.454346 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:45.454308    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" path="/var/lib/kubelet/pods/779a3756-a8ca-4057-94f0-6a217e8cb942/volumes"
Jul 09 18:17:49.255227 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:49.255193    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:17:49.255662 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:49.255638    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:17:59.256693 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:17:59.256607    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:18:09.256618 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:09.256573    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:18:16.895636 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895598    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895880    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container"
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895891    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container"
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895904    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kube-rbac-proxy"
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kube-rbac-proxy"
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895962    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kube-rbac-proxy"
Jul 09 18:18:16.896131 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.895971    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="779a3756-a8ca-4057-94f0-6a217e8cb942" containerName="kserve-container"
Jul 09 18:18:16.898122 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.898107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:16.900281 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.900260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8119-kube-rbac-proxy-sar-config\""
Jul 09 18:18:16.900389 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.900260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e8119-predictor-serving-cert\""
Jul 09 18:18:16.907961 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:16.907941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:18:17.015706 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.015669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.015901 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.015751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.015901 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.015779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7bf\" (UniqueName: \"kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.116336 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.116302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.116336 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.116340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7bf\" (UniqueName: \"kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.116577 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.116384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.117007 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.116986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.118676 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.118655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.124877 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.124858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7bf\" (UniqueName: \"kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf\") pod \"error-404-isvc-e8119-predictor-66cfd85494-sjkv5\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") " pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.208853 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.208763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:17.339965 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.339927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:18:17.342527 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:18:17.342488    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf59849_b7db_4dda_a979_7757170a151e.slice/crio-ccd2073f252dce5a680d5b90ffaa338fffb7564773c3935a432a0c1eb98a055c WatchSource:0}: Error finding container ccd2073f252dce5a680d5b90ffaa338fffb7564773c3935a432a0c1eb98a055c: Status 404 returned error can't find the container with id ccd2073f252dce5a680d5b90ffaa338fffb7564773c3935a432a0c1eb98a055c
Jul 09 18:18:17.379090 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:17.379057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerStarted","Data":"ccd2073f252dce5a680d5b90ffaa338fffb7564773c3935a432a0c1eb98a055c"}
Jul 09 18:18:18.384126 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.384089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerStarted","Data":"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"}
Jul 09 18:18:18.384624 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.384151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerStarted","Data":"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"}
Jul 09 18:18:18.384624 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.384185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:18.384624 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.384213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:18.385611 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.385582    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:18:18.404066 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:18.404019    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podStartSLOduration=2.40400493 podStartE2EDuration="2.40400493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:18.402548777 +0000 UTC m=+829.504075830" watchObservedRunningTime="2026-07-09 18:18:18.40400493 +0000 UTC m=+829.505532000"
Jul 09 18:18:19.256659 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:19.256624    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 09 18:18:19.388343 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:19.388307    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:18:24.393478 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:24.393450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:18:24.393934 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:24.393900    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:18:29.256552 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:29.256521    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:18:34.394103 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:34.394064    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:18:44.394544 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:44.394504    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:18:54.394242 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:18:54.394201    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:19:04.394866 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:19:04.394839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:19:29.331782 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:19:29.331687    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:19:29.335003 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:19:29.334981    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:19:29.336315 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:19:29.336296    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:19:29.339164 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:19:29.339145    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:24:29.355320 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:24:29.355291    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:24:29.359257 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:24:29.359238    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:24:29.359434 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:24:29.359418    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:24:29.363133 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:24:29.363116    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:26:55.782652 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.782563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:26:55.783238 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.782857    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container" containerID="cri-o://b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec" gracePeriod=30
Jul 09 18:26:55.783238 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.782862    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kube-rbac-proxy" containerID="cri-o://718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb" gracePeriod=30
Jul 09 18:26:55.810457 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.810424    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:26:55.813654 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.813638    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:55.815869 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.815846    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-64fa7-kube-rbac-proxy-sar-config\""
Jul 09 18:26:55.817819 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.817801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-64fa7-predictor-serving-cert\""
Jul 09 18:26:55.825617 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:55.825597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:26:56.001210 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.001174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbg6m\" (UniqueName: \"kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.001210 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.001217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.001433 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.001266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.101895 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.101863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.102072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.101917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbg6m\" (UniqueName: \"kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.102072 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.101948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.102072 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:26:56.102012    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-serving-cert: secret "error-404-isvc-64fa7-predictor-serving-cert" not found
Jul 09 18:26:56.102219 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:26:56.102092    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls podName:8fd12112-3b28-46fa-bcde-12c64d9f65e8 nodeName:}" failed. No retries permitted until 2026-07-09 18:26:56.602070122 +0000 UTC m=+1347.703597129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls") pod "error-404-isvc-64fa7-predictor-6678776cd-f6scb" (UID: "8fd12112-3b28-46fa-bcde-12c64d9f65e8") : secret "error-404-isvc-64fa7-predictor-serving-cert" not found
Jul 09 18:26:56.102638 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.102619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.114094 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.114069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbg6m\" (UniqueName: \"kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.301589 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.301558    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerID="718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb" exitCode=2
Jul 09 18:26:56.301777 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.301613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerDied","Data":"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"}
Jul 09 18:26:56.607730 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.607693    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.610127 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.610101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") pod \"error-404-isvc-64fa7-predictor-6678776cd-f6scb\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") " pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.722974 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.722936    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:56.853363 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:56.853327    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:26:56.856346 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:26:56.856324    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd12112_3b28_46fa_bcde_12c64d9f65e8.slice/crio-3519d992ddcee9715af0edeb883be21d54f0126f8de1458f0eae2d5f517e32b6 WatchSource:0}: Error finding container 3519d992ddcee9715af0edeb883be21d54f0126f8de1458f0eae2d5f517e32b6: Status 404 returned error can't find the container with id 3519d992ddcee9715af0edeb883be21d54f0126f8de1458f0eae2d5f517e32b6
Jul 09 18:26:57.307302 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.307274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerStarted","Data":"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"}
Jul 09 18:26:57.307446 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.307327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerStarted","Data":"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"}
Jul 09 18:26:57.307446 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.307353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:57.307446 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.307373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerStarted","Data":"3519d992ddcee9715af0edeb883be21d54f0126f8de1458f0eae2d5f517e32b6"}
Jul 09 18:26:57.307446 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.307411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:26:57.308699 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.308675    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:26:57.325440 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:57.325383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podStartSLOduration=2.325366635 podStartE2EDuration="2.325366635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:26:57.32420789 +0000 UTC m=+1348.425734937" watchObservedRunningTime="2026-07-09 18:26:57.325366635 +0000 UTC m=+1348.426893662"
Jul 09 18:26:58.311666 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:58.311626    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:26:59.030522 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.030497    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:26:59.227020 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.226930    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls\") pod \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") "
Jul 09 18:26:59.227020 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.226984    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59nwg\" (UniqueName: \"kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg\") pod \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") "
Jul 09 18:26:59.227020 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.227010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config\") pod \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\" (UID: \"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8\") "
Jul 09 18:26:59.227474 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.227447    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" (UID: "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:26:59.228979 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.228955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" (UID: "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:26:59.229074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.229043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg" (OuterVolumeSpecName: "kube-api-access-59nwg") pod "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" (UID: "1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8"). InnerVolumeSpecName "kube-api-access-59nwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:26:59.317089 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.317055    2561 generic.go:356] "Generic (PLEG): container finished" podID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerID="b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec" exitCode=0
Jul 09 18:26:59.317600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.317100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerDied","Data":"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"}
Jul 09 18:26:59.317600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.317134    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"
Jul 09 18:26:59.317600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.317173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g" event={"ID":"1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8","Type":"ContainerDied","Data":"8a712afab6e1c77abe9464a6ccf0c9a774d38b5cc876fa0026ff550e19955ce5"}
Jul 09 18:26:59.317600 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.317200    2561 scope.go:117] "RemoveContainer" containerID="718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"
Jul 09 18:26:59.326190 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.326085    2561 scope.go:117] "RemoveContainer" containerID="b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"
Jul 09 18:26:59.328540 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.328515    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:26:59.328540 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.328540    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59nwg\" (UniqueName: \"kubernetes.io/projected/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-api-access-59nwg\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:26:59.328711 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.328550    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:26:59.333703 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.333688    2561 scope.go:117] "RemoveContainer" containerID="718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"
Jul 09 18:26:59.333965 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:26:59.333944    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb\": container with ID starting with 718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb not found: ID does not exist" containerID="718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"
Jul 09 18:26:59.334038 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.333974    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb"} err="failed to get container status \"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb\": rpc error: code = NotFound desc = could not find container \"718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb\": container with ID starting with 718aaaf83f6b69249f00b941ed21a6a9ba0e63c9e774e20d6453a8ba228d39bb not found: ID does not exist"
Jul 09 18:26:59.334038 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.333991    2561 scope.go:117] "RemoveContainer" containerID="b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"
Jul 09 18:26:59.334197 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:26:59.334183    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec\": container with ID starting with b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec not found: ID does not exist" containerID="b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"
Jul 09 18:26:59.334233 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.334203    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec"} err="failed to get container status \"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec\": rpc error: code = NotFound desc = could not find container \"b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec\": container with ID starting with b1ed65c1623d28c392d2917d13d315d9572f9ae82adc64d836359e6885385cec not found: ID does not exist"
Jul 09 18:26:59.339403 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.339379    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:26:59.343569 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.343544    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a7bf-predictor-58d98848b6-gmt9g"]
Jul 09 18:26:59.454388 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:26:59.454354    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" path="/var/lib/kubelet/pods/1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8/volumes"
Jul 09 18:27:03.315963 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:03.315933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:27:03.316426 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:03.316397    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:27:13.317193 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:13.317149    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:27:23.316760 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:23.316677    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:27:31.659713 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:31.659681    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:27:31.660184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:31.659913    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" containerID="cri-o://77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f" gracePeriod=30
Jul 09 18:27:31.660184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:31.659968    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kube-rbac-proxy" containerID="cri-o://35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822" gracePeriod=30
Jul 09 18:27:32.024409 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024327    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:27:32.024651 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container"
Jul 09 18:27:32.024706 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container"
Jul 09 18:27:32.024706 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024666    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kube-rbac-proxy"
Jul 09 18:27:32.024706 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024671    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kube-rbac-proxy"
Jul 09 18:27:32.024842 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024714    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kserve-container"
Jul 09 18:27:32.024842 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.024740    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8aa7b4-3c06-4718-bf0c-24bf4ed449a8" containerName="kube-rbac-proxy"
Jul 09 18:27:32.027604 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.027589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.031130 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.031095    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dc46-predictor-serving-cert\""
Jul 09 18:27:32.031245 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.031226    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4dc46-kube-rbac-proxy-sar-config\""
Jul 09 18:27:32.040637 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.040612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:27:32.107818 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.107788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.107981 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.107851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.107981 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.107920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x625r\" (UniqueName: \"kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.209024 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.208993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.209165 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.209052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.209165 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.209077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x625r\" (UniqueName: \"kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.209274 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:27:32.209215    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-serving-cert: secret "error-404-isvc-4dc46-predictor-serving-cert" not found
Jul 09 18:27:32.209309 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:27:32.209281    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls podName:ca8bf240-a966-4b67-9c99-d3e9e5a9c02d nodeName:}" failed. No retries permitted until 2026-07-09 18:27:32.709263704 +0000 UTC m=+1383.810790709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls") pod "error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" (UID: "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d") : secret "error-404-isvc-4dc46-predictor-serving-cert" not found
Jul 09 18:27:32.209635 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.209617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.226870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.226839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x625r\" (UniqueName: \"kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.448960 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.448926    2561 generic.go:356] "Generic (PLEG): container finished" podID="4cf59849-b7db-4dda-a979-7757170a151e" containerID="35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822" exitCode=2
Jul 09 18:27:32.449121 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.448973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerDied","Data":"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"}
Jul 09 18:27:32.714320 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.714206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.716554 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.716528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") pod \"error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") " pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:32.937637 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:32.937600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:33.068605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.068579    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:27:33.071286 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:27:33.071255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8bf240_a966_4b67_9c99_d3e9e5a9c02d.slice/crio-43851ed400fd72b7d27715b575d35a815894dd1244be3d1da73acbbad16c2a2f WatchSource:0}: Error finding container 43851ed400fd72b7d27715b575d35a815894dd1244be3d1da73acbbad16c2a2f: Status 404 returned error can't find the container with id 43851ed400fd72b7d27715b575d35a815894dd1244be3d1da73acbbad16c2a2f
Jul 09 18:27:33.316550 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.316504    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 09 18:27:33.454613 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.454576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerStarted","Data":"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"}
Jul 09 18:27:33.454798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.454643    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:33.454798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.454673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerStarted","Data":"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"}
Jul 09 18:27:33.454798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.454702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:33.454798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.454753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerStarted","Data":"43851ed400fd72b7d27715b575d35a815894dd1244be3d1da73acbbad16c2a2f"}
Jul 09 18:27:33.455972 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.455944    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:27:33.474448 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:33.474403    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podStartSLOduration=1.474386231 podStartE2EDuration="1.474386231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:33.472683401 +0000 UTC m=+1384.574210446" watchObservedRunningTime="2026-07-09 18:27:33.474386231 +0000 UTC m=+1384.575913258"
Jul 09 18:27:34.389308 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:34.389258    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 09 18:27:34.394707 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:34.394682    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 09 18:27:34.458985 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:34.458948    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:27:34.904856 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:34.904829    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:27:35.035598 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.035509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls\") pod \"4cf59849-b7db-4dda-a979-7757170a151e\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") "
Jul 09 18:27:35.035769 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.035608    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config\") pod \"4cf59849-b7db-4dda-a979-7757170a151e\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") "
Jul 09 18:27:35.035769 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.035632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw7bf\" (UniqueName: \"kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf\") pod \"4cf59849-b7db-4dda-a979-7757170a151e\" (UID: \"4cf59849-b7db-4dda-a979-7757170a151e\") "
Jul 09 18:27:35.036084 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.036056    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cf59849-b7db-4dda-a979-7757170a151e" (UID: "4cf59849-b7db-4dda-a979-7757170a151e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:27:35.037738 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.037693    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cf59849-b7db-4dda-a979-7757170a151e" (UID: "4cf59849-b7db-4dda-a979-7757170a151e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:27:35.037738 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.037698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf" (OuterVolumeSpecName: "kube-api-access-cw7bf") pod "4cf59849-b7db-4dda-a979-7757170a151e" (UID: "4cf59849-b7db-4dda-a979-7757170a151e"). InnerVolumeSpecName "kube-api-access-cw7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:27:35.136468 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.136427    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cf59849-b7db-4dda-a979-7757170a151e-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:27:35.136468 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.136463    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cf59849-b7db-4dda-a979-7757170a151e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:27:35.136672 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.136478    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7bf\" (UniqueName: \"kubernetes.io/projected/4cf59849-b7db-4dda-a979-7757170a151e-kube-api-access-cw7bf\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:27:35.463855 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.463826    2561 generic.go:356] "Generic (PLEG): container finished" podID="4cf59849-b7db-4dda-a979-7757170a151e" containerID="77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f" exitCode=0
Jul 09 18:27:35.464215 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.463866    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"
Jul 09 18:27:35.464215 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.463906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerDied","Data":"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"}
Jul 09 18:27:35.464215 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.463957    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5" event={"ID":"4cf59849-b7db-4dda-a979-7757170a151e","Type":"ContainerDied","Data":"ccd2073f252dce5a680d5b90ffaa338fffb7564773c3935a432a0c1eb98a055c"}
Jul 09 18:27:35.464215 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.463983    2561 scope.go:117] "RemoveContainer" containerID="35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"
Jul 09 18:27:35.472257 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.472246    2561 scope.go:117] "RemoveContainer" containerID="77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"
Jul 09 18:27:35.479384 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.479356    2561 scope.go:117] "RemoveContainer" containerID="35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"
Jul 09 18:27:35.480415 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:27:35.480388    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822\": container with ID starting with 35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822 not found: ID does not exist" containerID="35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"
Jul 09 18:27:35.480523 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.480422    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822"} err="failed to get container status \"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822\": rpc error: code = NotFound desc = could not find container \"35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822\": container with ID starting with 35a5e128b2e4c0e2076a16ea0219f5e1559bef143c932ac1cdb4a9bf15c52822 not found: ID does not exist"
Jul 09 18:27:35.480523 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.480441    2561 scope.go:117] "RemoveContainer" containerID="77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"
Jul 09 18:27:35.480999 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:27:35.480716    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f\": container with ID starting with 77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f not found: ID does not exist" containerID="77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"
Jul 09 18:27:35.481044 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.481007    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f"} err="failed to get container status \"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f\": rpc error: code = NotFound desc = could not find container \"77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f\": container with ID starting with 77465de6fd370018c9ad1ec38c46a075e5c9640b82c5778b90e02ac654bf337f not found: ID does not exist"
Jul 09 18:27:35.481419 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.481391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:27:35.485123 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:35.485099    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e8119-predictor-66cfd85494-sjkv5"]
Jul 09 18:27:37.458992 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:37.458959    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf59849-b7db-4dda-a979-7757170a151e" path="/var/lib/kubelet/pods/4cf59849-b7db-4dda-a979-7757170a151e/volumes"
Jul 09 18:27:39.463962 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:39.463937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:27:39.464439 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:39.464412    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:27:43.316875 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:43.316849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:27:49.464369 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:49.464329    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:27:59.464960 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:27:59.464921    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:28:05.985956 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:05.985924    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:28:05.986460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:05.986216    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container" containerID="cri-o://92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0" gracePeriod=30
Jul 09 18:28:05.986460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:05.986288    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kube-rbac-proxy" containerID="cri-o://438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d" gracePeriod=30
Jul 09 18:28:06.230297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230266    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:28:06.230563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container"
Jul 09 18:28:06.230615 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230565    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container"
Jul 09 18:28:06.230615 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230572    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kube-rbac-proxy"
Jul 09 18:28:06.230615 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230577    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kube-rbac-proxy"
Jul 09 18:28:06.230718 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230631    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kube-rbac-proxy"
Jul 09 18:28:06.230718 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.230638    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf59849-b7db-4dda-a979-7757170a151e" containerName="kserve-container"
Jul 09 18:28:06.232763 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.232747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.235480 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.235463    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f76a1-predictor-serving-cert\""
Jul 09 18:28:06.235882 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.235868    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f76a1-kube-rbac-proxy-sar-config\""
Jul 09 18:28:06.244284 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.244260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:28:06.287070 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.287038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.287235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.287089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nj4\" (UniqueName: \"kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.287235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.287110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.388423 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.388369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.388610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.388461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nj4\" (UniqueName: \"kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.388610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.388495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.388610 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:06.388540    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-serving-cert: secret "error-404-isvc-f76a1-predictor-serving-cert" not found
Jul 09 18:28:06.388610 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:06.388609    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls podName:5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063 nodeName:}" failed. No retries permitted until 2026-07-09 18:28:06.888586958 +0000 UTC m=+1417.990113963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls") pod "error-404-isvc-f76a1-predictor-55d9479568-xcv9f" (UID: "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063") : secret "error-404-isvc-f76a1-predictor-serving-cert" not found
Jul 09 18:28:06.389267 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.389242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.399549 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.399526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nj4\" (UniqueName: \"kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.584123 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.584091    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerID="438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d" exitCode=2
Jul 09 18:28:06.584300 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.584153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerDied","Data":"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"}
Jul 09 18:28:06.893930 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.893842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:06.896255 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:06.896231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") pod \"error-404-isvc-f76a1-predictor-55d9479568-xcv9f\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") " pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:07.142649 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.142614    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:07.275607 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.275585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:28:07.277821 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:28:07.277792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4a1a1e_fe9b_4bcb_8bec_571bae6c7063.slice/crio-12592389c7725013b3dfdcefc578ebf41dd7976f19b145549d2c2f62dbd2df62 WatchSource:0}: Error finding container 12592389c7725013b3dfdcefc578ebf41dd7976f19b145549d2c2f62dbd2df62: Status 404 returned error can't find the container with id 12592389c7725013b3dfdcefc578ebf41dd7976f19b145549d2c2f62dbd2df62
Jul 09 18:28:07.590151 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.590110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerStarted","Data":"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"}
Jul 09 18:28:07.590316 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.590178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerStarted","Data":"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"}
Jul 09 18:28:07.590316 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.590199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerStarted","Data":"12592389c7725013b3dfdcefc578ebf41dd7976f19b145549d2c2f62dbd2df62"}
Jul 09 18:28:07.590316 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.590222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:07.612174 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:07.612130    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podStartSLOduration=1.612116446 podStartE2EDuration="1.612116446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:07.609296917 +0000 UTC m=+1418.710823938" watchObservedRunningTime="2026-07-09 18:28:07.612116446 +0000 UTC m=+1418.713643470"
Jul 09 18:28:08.312206 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:08.312166    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 09 18:28:08.594790 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:08.594674    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:08.596066 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:08.596035    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:09.230759 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.230736    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:28:09.314108 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.314078    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config\") pod \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") "
Jul 09 18:28:09.314563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.314148    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbg6m\" (UniqueName: \"kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m\") pod \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") "
Jul 09 18:28:09.314563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.314249    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") pod \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\" (UID: \"8fd12112-3b28-46fa-bcde-12c64d9f65e8\") "
Jul 09 18:28:09.314563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.314492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8fd12112-3b28-46fa-bcde-12c64d9f65e8" (UID: "8fd12112-3b28-46fa-bcde-12c64d9f65e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:09.316205 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.316181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m" (OuterVolumeSpecName: "kube-api-access-lbg6m") pod "8fd12112-3b28-46fa-bcde-12c64d9f65e8" (UID: "8fd12112-3b28-46fa-bcde-12c64d9f65e8"). InnerVolumeSpecName "kube-api-access-lbg6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:09.316609 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.316593    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fd12112-3b28-46fa-bcde-12c64d9f65e8" (UID: "8fd12112-3b28-46fa-bcde-12c64d9f65e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:09.415787 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.415740    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbg6m\" (UniqueName: \"kubernetes.io/projected/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-api-access-lbg6m\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.415787 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.415771    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fd12112-3b28-46fa-bcde-12c64d9f65e8-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.415787 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.415795    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fd12112-3b28-46fa-bcde-12c64d9f65e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:09.464965 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.464924    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 09 18:28:09.599558 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.599466    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerID="92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0" exitCode=0
Jul 09 18:28:09.599558 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.599518    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"
Jul 09 18:28:09.599805 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.599550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerDied","Data":"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"}
Jul 09 18:28:09.599805 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.599616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb" event={"ID":"8fd12112-3b28-46fa-bcde-12c64d9f65e8","Type":"ContainerDied","Data":"3519d992ddcee9715af0edeb883be21d54f0126f8de1458f0eae2d5f517e32b6"}
Jul 09 18:28:09.599805 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.599646    2561 scope.go:117] "RemoveContainer" containerID="438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"
Jul 09 18:28:09.600252 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.600229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:09.607475 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.607457    2561 scope.go:117] "RemoveContainer" containerID="92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"
Jul 09 18:28:09.614954 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.614932    2561 scope.go:117] "RemoveContainer" containerID="438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"
Jul 09 18:28:09.615262 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:09.615240    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d\": container with ID starting with 438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d not found: ID does not exist" containerID="438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"
Jul 09 18:28:09.615367 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.615267    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d"} err="failed to get container status \"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d\": rpc error: code = NotFound desc = could not find container \"438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d\": container with ID starting with 438430c28a2fbd2ca49a00f2bef7c71d8164e924567baf93510328277b36fb9d not found: ID does not exist"
Jul 09 18:28:09.615367 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.615283    2561 scope.go:117] "RemoveContainer" containerID="92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"
Jul 09 18:28:09.615545 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:09.615529    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0\": container with ID starting with 92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0 not found: ID does not exist" containerID="92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"
Jul 09 18:28:09.615606 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.615547    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0"} err="failed to get container status \"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0\": rpc error: code = NotFound desc = could not find container \"92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0\": container with ID starting with 92f91cf1261f39418b92216285fb30b752491f1592114e9e217fb2877e5a04b0 not found: ID does not exist"
Jul 09 18:28:09.618677 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.618651    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:28:09.622527 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:09.622504    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64fa7-predictor-6678776cd-f6scb"]
Jul 09 18:28:11.454009 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:11.453979    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" path="/var/lib/kubelet/pods/8fd12112-3b28-46fa-bcde-12c64d9f65e8/volumes"
Jul 09 18:28:14.604832 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:14.604806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:28:14.605299 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:14.605272    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:19.464893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:19.464866    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:28:24.606165 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:24.606074    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:34.606144 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:34.606099    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:42.071888 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.071855    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:28:42.072384 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.072090    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container" containerID="cri-o://5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a" gracePeriod=30
Jul 09 18:28:42.072384 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.072142    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kube-rbac-proxy" containerID="cri-o://e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438" gracePeriod=30
Jul 09 18:28:42.411300 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411218    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:28:42.411551 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411539    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kube-rbac-proxy"
Jul 09 18:28:42.411592 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411554    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kube-rbac-proxy"
Jul 09 18:28:42.411592 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411585    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container"
Jul 09 18:28:42.411592 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411591    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container"
Jul 09 18:28:42.411688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411637    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kube-rbac-proxy"
Jul 09 18:28:42.411688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.411647    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd12112-3b28-46fa-bcde-12c64d9f65e8" containerName="kserve-container"
Jul 09 18:28:42.414696 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.414679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.418636 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.418613    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2ff98-kube-rbac-proxy-sar-config\""
Jul 09 18:28:42.421246 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.421222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2ff98-predictor-serving-cert\""
Jul 09 18:28:42.433770 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.433742    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:28:42.494790 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.494752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.494989 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.494804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztqr\" (UniqueName: \"kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.494989 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.494903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.595281 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.595241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.595475 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.595296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.595475 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.595341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztqr\" (UniqueName: \"kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.595475 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:42.595457    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-serving-cert: secret "error-404-isvc-2ff98-predictor-serving-cert" not found
Jul 09 18:28:42.595639 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:42.595534    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls podName:5294cfed-ae28-4fdd-823b-2ea8ca5e9171 nodeName:}" failed. No retries permitted until 2026-07-09 18:28:43.095514777 +0000 UTC m=+1454.197041791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls") pod "error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" (UID: "5294cfed-ae28-4fdd-823b-2ea8ca5e9171") : secret "error-404-isvc-2ff98-predictor-serving-cert" not found
Jul 09 18:28:42.596028 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.596005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.604546 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.604523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztqr\" (UniqueName: \"kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:42.734516 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.734437    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerID="e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438" exitCode=2
Jul 09 18:28:42.734658 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:42.734517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerDied","Data":"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"}
Jul 09 18:28:43.100268 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.100228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:43.102598 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.102579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") pod \"error-404-isvc-2ff98-predictor-7d578668c6-rdpgq\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") " pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:43.327061 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.327015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:43.462289 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.462149    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:28:43.464565 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:28:43.464542    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5294cfed_ae28_4fdd_823b_2ea8ca5e9171.slice/crio-0125eaf9ac223d24667459fbc7752da0e0b83bcdabd9536bd694adbdf0e7c136 WatchSource:0}: Error finding container 0125eaf9ac223d24667459fbc7752da0e0b83bcdabd9536bd694adbdf0e7c136: Status 404 returned error can't find the container with id 0125eaf9ac223d24667459fbc7752da0e0b83bcdabd9536bd694adbdf0e7c136
Jul 09 18:28:43.739818 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.739730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerStarted","Data":"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"}
Jul 09 18:28:43.739818 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.739785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerStarted","Data":"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"}
Jul 09 18:28:43.739818 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.739805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerStarted","Data":"0125eaf9ac223d24667459fbc7752da0e0b83bcdabd9536bd694adbdf0e7c136"}
Jul 09 18:28:43.740064 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.739887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:43.761276 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:43.761221    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podStartSLOduration=1.761204701 podStartE2EDuration="1.761204701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:43.759347723 +0000 UTC m=+1454.860874751" watchObservedRunningTime="2026-07-09 18:28:43.761204701 +0000 UTC m=+1454.862731728"
Jul 09 18:28:44.459366 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:44.459321    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 09 18:28:44.606207 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:44.606157    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 09 18:28:44.744200 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:44.744106    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:44.745602 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:44.745569    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:28:45.316444 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.316419    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:28:45.422444 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.422358    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") pod \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") "
Jul 09 18:28:45.422444 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.422430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config\") pod \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") "
Jul 09 18:28:45.422625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.422451    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x625r\" (UniqueName: \"kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r\") pod \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\" (UID: \"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d\") "
Jul 09 18:28:45.422810 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.422783    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" (UID: "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:45.424485 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.424453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" (UID: "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:45.424561 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.424525    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r" (OuterVolumeSpecName: "kube-api-access-x625r") pod "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" (UID: "ca8bf240-a966-4b67-9c99-d3e9e5a9c02d"). InnerVolumeSpecName "kube-api-access-x625r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:45.523583 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.523550    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:45.523583 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.523579    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x625r\" (UniqueName: \"kubernetes.io/projected/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-kube-api-access-x625r\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:45.523583 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.523589    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:28:45.755276 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755196    2561 generic.go:356] "Generic (PLEG): container finished" podID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerID="5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a" exitCode=0
Jul 09 18:28:45.755418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755274    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"
Jul 09 18:28:45.755418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerDied","Data":"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"}
Jul 09 18:28:45.755418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z" event={"ID":"ca8bf240-a966-4b67-9c99-d3e9e5a9c02d","Type":"ContainerDied","Data":"43851ed400fd72b7d27715b575d35a815894dd1244be3d1da73acbbad16c2a2f"}
Jul 09 18:28:45.755418 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755403    2561 scope.go:117] "RemoveContainer" containerID="e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"
Jul 09 18:28:45.755802 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.755766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:28:45.763890 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.763870    2561 scope.go:117] "RemoveContainer" containerID="5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"
Jul 09 18:28:45.770914 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.770898    2561 scope.go:117] "RemoveContainer" containerID="e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"
Jul 09 18:28:45.771136 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:45.771119    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438\": container with ID starting with e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438 not found: ID does not exist" containerID="e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"
Jul 09 18:28:45.771184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.771143    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438"} err="failed to get container status \"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438\": rpc error: code = NotFound desc = could not find container \"e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438\": container with ID starting with e4a8ceea0af4c0350114bdfb2fbf83a23e536b1e1a84b479d033266a934d0438 not found: ID does not exist"
Jul 09 18:28:45.771184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.771158    2561 scope.go:117] "RemoveContainer" containerID="5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"
Jul 09 18:28:45.771377 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:28:45.771356    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a\": container with ID starting with 5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a not found: ID does not exist" containerID="5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"
Jul 09 18:28:45.771437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.771386    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a"} err="failed to get container status \"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a\": rpc error: code = NotFound desc = could not find container \"5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a\": container with ID starting with 5509f0e7c4053a68aba6d4c195f8bfd81977e9e892de4f6340249bf95b13862a not found: ID does not exist"
Jul 09 18:28:45.801621 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.801588    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:28:45.806061 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:45.806037    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4dc46-predictor-569d6c6dd4-m7r8z"]
Jul 09 18:28:47.453889 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:47.453860    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" path="/var/lib/kubelet/pods/ca8bf240-a966-4b67-9c99-d3e9e5a9c02d/volumes"
Jul 09 18:28:50.760030 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:50.759999    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:28:50.760605 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:50.760579    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:28:54.606366 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:28:54.606335    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:29:00.761229 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:00.761183    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:29:10.761040 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:10.761002    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:29:20.760872 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:20.760833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 09 18:29:29.377685 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:29.377653    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:29:29.381785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:29.381757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:29:29.382028 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:29.382012    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:29:29.386193 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:29.386174    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:29:30.761421 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:29:30.761380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:34:29.399825 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:34:29.399797    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:34:29.403898 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:34:29.403877    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:34:29.404472 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:34:29.404457    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:34:29.408592 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:34:29.408573    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:37:20.840750 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.840656    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:37:20.843107 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.840909    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container" containerID="cri-o://a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90" gracePeriod=30
Jul 09 18:37:20.843107 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.840935    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kube-rbac-proxy" containerID="cri-o://37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48" gracePeriod=30
Jul 09 18:37:20.944022 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.943986    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:37:20.944286 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944274    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container"
Jul 09 18:37:20.944344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944287    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container"
Jul 09 18:37:20.944344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944330    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kube-rbac-proxy"
Jul 09 18:37:20.944344 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944338    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kube-rbac-proxy"
Jul 09 18:37:20.944448 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944395    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kserve-container"
Jul 09 18:37:20.944448 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.944403    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca8bf240-a966-4b67-9c99-d3e9e5a9c02d" containerName="kube-rbac-proxy"
Jul 09 18:37:20.947079 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.947063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:20.949109 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.949087    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb3ff-kube-rbac-proxy-sar-config\""
Jul 09 18:37:20.949223 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.949201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb3ff-predictor-serving-cert\""
Jul 09 18:37:20.956308 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:20.956279    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:37:21.050060 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.050018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.050060 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.050062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgbdp\" (UniqueName: \"kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.050280 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.050144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.151433 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.151338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.151433 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.151399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.151433 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.151420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgbdp\" (UniqueName: \"kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.151713 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:37:21.151563    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-serving-cert: secret "error-404-isvc-eb3ff-predictor-serving-cert" not found
Jul 09 18:37:21.151713 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:37:21.151626    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls podName:3ac59db0-951f-49c0-aaa0-e3937fb552ee nodeName:}" failed. No retries permitted until 2026-07-09 18:37:21.651611492 +0000 UTC m=+1972.753138497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls") pod "error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" (UID: "3ac59db0-951f-49c0-aaa0-e3937fb552ee") : secret "error-404-isvc-eb3ff-predictor-serving-cert" not found
Jul 09 18:37:21.152123 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.152102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.160212 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.160191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgbdp\" (UniqueName: \"kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.650934 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.650905    2561 generic.go:356] "Generic (PLEG): container finished" podID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerID="37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48" exitCode=2
Jul 09 18:37:21.651105 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.650979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerDied","Data":"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"}
Jul 09 18:37:21.655291 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.655271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.657775 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.657745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") pod \"error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") " pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.858561 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.858521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:21.987372 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:21.987336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:37:21.991285 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:37:21.991261    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac59db0_951f_49c0_aaa0_e3937fb552ee.slice/crio-40826781b7c8202dcfb574b424e1861ac862b01553694f43c00d1ab8a187e57e WatchSource:0}: Error finding container 40826781b7c8202dcfb574b424e1861ac862b01553694f43c00d1ab8a187e57e: Status 404 returned error can't find the container with id 40826781b7c8202dcfb574b424e1861ac862b01553694f43c00d1ab8a187e57e
Jul 09 18:37:22.656067 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:22.656031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerStarted","Data":"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"}
Jul 09 18:37:22.656235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:22.656088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerStarted","Data":"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"}
Jul 09 18:37:22.656235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:22.656115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerStarted","Data":"40826781b7c8202dcfb574b424e1861ac862b01553694f43c00d1ab8a187e57e"}
Jul 09 18:37:22.656235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:22.656183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:22.672852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:22.672795    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podStartSLOduration=2.672778407 podStartE2EDuration="2.672778407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:22.672029164 +0000 UTC m=+1973.773556212" watchObservedRunningTime="2026-07-09 18:37:22.672778407 +0000 UTC m=+1973.774305433"
Jul 09 18:37:23.659671 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:23.659638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:23.660937 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:23.660911    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:23.999404 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:23.999376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:37:24.074431 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.074395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9nj4\" (UniqueName: \"kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4\") pod \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") "
Jul 09 18:37:24.074601 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.074455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config\") pod \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") "
Jul 09 18:37:24.074601 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.074519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") pod \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\" (UID: \"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063\") "
Jul 09 18:37:24.074929 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.074897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" (UID: "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:37:24.076455 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.076433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" (UID: "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:37:24.076568 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.076549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4" (OuterVolumeSpecName: "kube-api-access-k9nj4") pod "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" (UID: "5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063"). InnerVolumeSpecName "kube-api-access-k9nj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:37:24.175735 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.175694    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.175898 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.175750    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nj4\" (UniqueName: \"kubernetes.io/projected/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-api-access-k9nj4\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.175898 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.175761    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:37:24.664998 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.664965    2561 generic.go:356] "Generic (PLEG): container finished" podID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerID="a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90" exitCode=0
Jul 09 18:37:24.665460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.665011    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"
Jul 09 18:37:24.665460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.665052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerDied","Data":"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"}
Jul 09 18:37:24.665460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.665127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f" event={"ID":"5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063","Type":"ContainerDied","Data":"12592389c7725013b3dfdcefc578ebf41dd7976f19b145549d2c2f62dbd2df62"}
Jul 09 18:37:24.665460 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.665169    2561 scope.go:117] "RemoveContainer" containerID="37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"
Jul 09 18:37:24.665692 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.665671    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:24.673463 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.673449    2561 scope.go:117] "RemoveContainer" containerID="a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"
Jul 09 18:37:24.680382 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.680362    2561 scope.go:117] "RemoveContainer" containerID="37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"
Jul 09 18:37:24.680637 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:37:24.680619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48\": container with ID starting with 37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48 not found: ID does not exist" containerID="37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"
Jul 09 18:37:24.680688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.680645    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48"} err="failed to get container status \"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48\": rpc error: code = NotFound desc = could not find container \"37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48\": container with ID starting with 37ca077fb964d9d70afccac8d90c5409c0c2de2bcb5b545460cc78ef294d1b48 not found: ID does not exist"
Jul 09 18:37:24.680688 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.680661    2561 scope.go:117] "RemoveContainer" containerID="a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"
Jul 09 18:37:24.680968 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:37:24.680942    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90\": container with ID starting with a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90 not found: ID does not exist" containerID="a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"
Jul 09 18:37:24.681019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.680976    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90"} err="failed to get container status \"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90\": rpc error: code = NotFound desc = could not find container \"a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90\": container with ID starting with a8edc538e972e2b39b61578c14c0703486c636c810c9439a49b51f9fdca6cc90 not found: ID does not exist"
Jul 09 18:37:24.684682 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.684658    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:37:24.688666 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:24.688648    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f76a1-predictor-55d9479568-xcv9f"]
Jul 09 18:37:25.454360 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:25.454330    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" path="/var/lib/kubelet/pods/5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063/volumes"
Jul 09 18:37:29.670019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:29.669989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:37:29.670498 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:29.670473    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:39.671437 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:39.671396    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:49.670659 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:49.670614    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:57.096571 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.096536    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:37:57.097092 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.096877    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container" containerID="cri-o://b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104" gracePeriod=30
Jul 09 18:37:57.097092 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.096897    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kube-rbac-proxy" containerID="cri-o://f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c" gracePeriod=30
Jul 09 18:37:57.232347 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232313    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:37:57.232674 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232657    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kube-rbac-proxy"
Jul 09 18:37:57.232785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232676    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kube-rbac-proxy"
Jul 09 18:37:57.232785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232696    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container"
Jul 09 18:37:57.232785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232705    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container"
Jul 09 18:37:57.232951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232797    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kserve-container"
Jul 09 18:37:57.232951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.232811    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4a1a1e-fe9b-4bcb-8bec-571bae6c7063" containerName="kube-rbac-proxy"
Jul 09 18:37:57.235738 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.235708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.237886 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.237864    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bed12-predictor-serving-cert\""
Jul 09 18:37:57.237972 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.237870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bed12-kube-rbac-proxy-sar-config\""
Jul 09 18:37:57.245925 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.245905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:37:57.356221 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.356132    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.356221 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.356178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.356400 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.356273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gm2\" (UniqueName: \"kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.460564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.458767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gm2\" (UniqueName: \"kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.460564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.458874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.460564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.458919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.460564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.460046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.463999 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.462299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.467196 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.467168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gm2\" (UniqueName: \"kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2\") pod \"error-404-isvc-bed12-predictor-646d855c94-8pr67\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") " pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.548677 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.548639    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:57.684394 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.684368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:37:57.686711 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:37:57.686689    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f082334_acba_45d2_885e_ba1fe7db0a74.slice/crio-41538fd887fbaf4bd2a26be00bc158a2b62b69f9b62f7b2d9da20e8b7d883473 WatchSource:0}: Error finding container 41538fd887fbaf4bd2a26be00bc158a2b62b69f9b62f7b2d9da20e8b7d883473: Status 404 returned error can't find the container with id 41538fd887fbaf4bd2a26be00bc158a2b62b69f9b62f7b2d9da20e8b7d883473
Jul 09 18:37:57.790761 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.790712    2561 generic.go:356] "Generic (PLEG): container finished" podID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerID="f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c" exitCode=2
Jul 09 18:37:57.790874 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.790754    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerDied","Data":"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"}
Jul 09 18:37:57.792174 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.792153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerStarted","Data":"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"}
Jul 09 18:37:57.792272 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:57.792205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerStarted","Data":"41538fd887fbaf4bd2a26be00bc158a2b62b69f9b62f7b2d9da20e8b7d883473"}
Jul 09 18:37:58.797351 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:58.797312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerStarted","Data":"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"}
Jul 09 18:37:58.797822 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:58.797459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:58.814374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:58.814325    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podStartSLOduration=1.814305797 podStartE2EDuration="1.814305797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:58.813490946 +0000 UTC m=+2009.915017998" watchObservedRunningTime="2026-07-09 18:37:58.814305797 +0000 UTC m=+2009.915832825"
Jul 09 18:37:59.670448 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:59.670411    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 18:37:59.801376 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:59.801340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:37:59.802972 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:37:59.802939    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:00.445531 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.445506    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:38:00.481833 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.481753    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config\") pod \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") "
Jul 09 18:38:00.481833 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.481792    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fztqr\" (UniqueName: \"kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr\") pod \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") "
Jul 09 18:38:00.481833 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.481822    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") pod \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\" (UID: \"5294cfed-ae28-4fdd-823b-2ea8ca5e9171\") "
Jul 09 18:38:00.482180 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.482152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5294cfed-ae28-4fdd-823b-2ea8ca5e9171" (UID: "5294cfed-ae28-4fdd-823b-2ea8ca5e9171"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:38:00.483912 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.483883    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5294cfed-ae28-4fdd-823b-2ea8ca5e9171" (UID: "5294cfed-ae28-4fdd-823b-2ea8ca5e9171"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:00.484000 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.483943    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr" (OuterVolumeSpecName: "kube-api-access-fztqr") pod "5294cfed-ae28-4fdd-823b-2ea8ca5e9171" (UID: "5294cfed-ae28-4fdd-823b-2ea8ca5e9171"). InnerVolumeSpecName "kube-api-access-fztqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:00.582272 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.582235    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:00.582272 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.582267    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztqr\" (UniqueName: \"kubernetes.io/projected/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-kube-api-access-fztqr\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:00.582272 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.582277    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5294cfed-ae28-4fdd-823b-2ea8ca5e9171-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:00.806077 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806046    2561 generic.go:356] "Generic (PLEG): container finished" podID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerID="b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104" exitCode=0
Jul 09 18:38:00.806542 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806103    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"
Jul 09 18:38:00.806542 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerDied","Data":"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"}
Jul 09 18:38:00.806542 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq" event={"ID":"5294cfed-ae28-4fdd-823b-2ea8ca5e9171","Type":"ContainerDied","Data":"0125eaf9ac223d24667459fbc7752da0e0b83bcdabd9536bd694adbdf0e7c136"}
Jul 09 18:38:00.806542 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806216    2561 scope.go:117] "RemoveContainer" containerID="f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"
Jul 09 18:38:00.806836 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.806774    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:00.814816 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.814797    2561 scope.go:117] "RemoveContainer" containerID="b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"
Jul 09 18:38:00.822370 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.822342    2561 scope.go:117] "RemoveContainer" containerID="f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"
Jul 09 18:38:00.822681 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:38:00.822662    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c\": container with ID starting with f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c not found: ID does not exist" containerID="f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"
Jul 09 18:38:00.822784 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.822692    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c"} err="failed to get container status \"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c\": rpc error: code = NotFound desc = could not find container \"f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c\": container with ID starting with f358c3dd8baf0031cfef6db8ae282a54011e91def82b589f89dd4c007f1ab74c not found: ID does not exist"
Jul 09 18:38:00.822784 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.822710    2561 scope.go:117] "RemoveContainer" containerID="b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"
Jul 09 18:38:00.823000 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:38:00.822985    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104\": container with ID starting with b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104 not found: ID does not exist" containerID="b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"
Jul 09 18:38:00.823037 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.823004    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104"} err="failed to get container status \"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104\": rpc error: code = NotFound desc = could not find container \"b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104\": container with ID starting with b2641b5bc252c06c09408b4b261a9a4af573ef9d1551a862774ddd23d15ff104 not found: ID does not exist"
Jul 09 18:38:00.828563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.828537    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:38:00.832686 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:00.832662    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2ff98-predictor-7d578668c6-rdpgq"]
Jul 09 18:38:01.454458 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:01.454424    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" path="/var/lib/kubelet/pods/5294cfed-ae28-4fdd-823b-2ea8ca5e9171/volumes"
Jul 09 18:38:05.810761 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:05.810714    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:38:05.811253 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:05.811228    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:09.671574 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:09.671548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:38:15.812019 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:15.811980    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:25.811613 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:25.811570    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:31.275263 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.275234    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:38:31.275679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.275506    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container" containerID="cri-o://ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac" gracePeriod=30
Jul 09 18:38:31.275679 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.275522    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kube-rbac-proxy" containerID="cri-o://b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce" gracePeriod=30
Jul 09 18:38:31.445294 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445261    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:38:31.445564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container"
Jul 09 18:38:31.445564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445564    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container"
Jul 09 18:38:31.445663 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445574    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kube-rbac-proxy"
Jul 09 18:38:31.445663 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445579    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kube-rbac-proxy"
Jul 09 18:38:31.445663 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445636    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kserve-container"
Jul 09 18:38:31.445663 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.445644    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5294cfed-ae28-4fdd-823b-2ea8ca5e9171" containerName="kube-rbac-proxy"
Jul 09 18:38:31.448650 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.448630    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.450965 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.450945    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-865f9-predictor-serving-cert\""
Jul 09 18:38:31.450965 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.450958    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-865f9-kube-rbac-proxy-sar-config\""
Jul 09 18:38:31.456513 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.456491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:38:31.539704 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.539614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.539704 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.539666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.539983 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.539768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hf9b\" (UniqueName: \"kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.640654 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.640614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hf9b\" (UniqueName: \"kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.640798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.640761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.640798 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.640793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.641511 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.641491    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.643132 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.643111    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.649306 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.649286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hf9b\" (UniqueName: \"kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b\") pod \"error-404-isvc-865f9-predictor-6f895b7954-9lbzc\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") " pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.759526 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.759495    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:31.883564 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.883540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:38:31.885849 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:38:31.885824    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268aa6f6_f9b0_4244_80c4_0b21f3590cb3.slice/crio-886a1503b1f6c2c2b96da1fe2e7e559d8e051f49cee35cdeae8d56c89cf739b1 WatchSource:0}: Error finding container 886a1503b1f6c2c2b96da1fe2e7e559d8e051f49cee35cdeae8d56c89cf739b1: Status 404 returned error can't find the container with id 886a1503b1f6c2c2b96da1fe2e7e559d8e051f49cee35cdeae8d56c89cf739b1
Jul 09 18:38:31.923832 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.923799    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerID="b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce" exitCode=2
Jul 09 18:38:31.923936 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.923870    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerDied","Data":"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"}
Jul 09 18:38:31.925111 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:31.925090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerStarted","Data":"886a1503b1f6c2c2b96da1fe2e7e559d8e051f49cee35cdeae8d56c89cf739b1"}
Jul 09 18:38:32.930322 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:32.930286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerStarted","Data":"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"}
Jul 09 18:38:32.930790 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:32.930349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerStarted","Data":"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"}
Jul 09 18:38:32.930790 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:32.930425    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:32.947217 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:32.947175    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podStartSLOduration=1.947159844 podStartE2EDuration="1.947159844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:32.946082275 +0000 UTC m=+2044.047609301" watchObservedRunningTime="2026-07-09 18:38:32.947159844 +0000 UTC m=+2044.048686870"
Jul 09 18:38:33.933667 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:33.933639    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:33.934923 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:33.934894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:38:34.625168 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.625144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:38:34.766797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.766688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config\") pod \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") "
Jul 09 18:38:34.766797 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.766773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") pod \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") "
Jul 09 18:38:34.767014 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.766827    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgbdp\" (UniqueName: \"kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp\") pod \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\" (UID: \"3ac59db0-951f-49c0-aaa0-e3937fb552ee\") "
Jul 09 18:38:34.767140 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.767115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ac59db0-951f-49c0-aaa0-e3937fb552ee" (UID: "3ac59db0-951f-49c0-aaa0-e3937fb552ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:38:34.768879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.768858    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp" (OuterVolumeSpecName: "kube-api-access-vgbdp") pod "3ac59db0-951f-49c0-aaa0-e3937fb552ee" (UID: "3ac59db0-951f-49c0-aaa0-e3937fb552ee"). InnerVolumeSpecName "kube-api-access-vgbdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:34.768879 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.768865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ac59db0-951f-49c0-aaa0-e3937fb552ee" (UID: "3ac59db0-951f-49c0-aaa0-e3937fb552ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:34.867519 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.867473    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgbdp\" (UniqueName: \"kubernetes.io/projected/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-api-access-vgbdp\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:34.867519 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.867513    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ac59db0-951f-49c0-aaa0-e3937fb552ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:34.867519 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.867523    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ac59db0-951f-49c0-aaa0-e3937fb552ee-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:38:34.938757 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.938705    2561 generic.go:356] "Generic (PLEG): container finished" podID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerID="ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac" exitCode=0
Jul 09 18:38:34.939135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.938781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"
Jul 09 18:38:34.939135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.938783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerDied","Data":"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"}
Jul 09 18:38:34.939135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.938853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd" event={"ID":"3ac59db0-951f-49c0-aaa0-e3937fb552ee","Type":"ContainerDied","Data":"40826781b7c8202dcfb574b424e1861ac862b01553694f43c00d1ab8a187e57e"}
Jul 09 18:38:34.939135 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.938891    2561 scope.go:117] "RemoveContainer" containerID="b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"
Jul 09 18:38:34.939625 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.939596    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:38:34.947122 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.947107    2561 scope.go:117] "RemoveContainer" containerID="ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"
Jul 09 18:38:34.954074 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.954060    2561 scope.go:117] "RemoveContainer" containerID="b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"
Jul 09 18:38:34.954325 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:38:34.954309    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce\": container with ID starting with b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce not found: ID does not exist" containerID="b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"
Jul 09 18:38:34.954377 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.954334    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce"} err="failed to get container status \"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce\": rpc error: code = NotFound desc = could not find container \"b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce\": container with ID starting with b3899a74d74990f3783b813368766393fab072c928bdfb3333f8a1b0425049ce not found: ID does not exist"
Jul 09 18:38:34.954377 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.954351    2561 scope.go:117] "RemoveContainer" containerID="ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"
Jul 09 18:38:34.954570 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:38:34.954552    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac\": container with ID starting with ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac not found: ID does not exist" containerID="ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"
Jul 09 18:38:34.954610 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.954573    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac"} err="failed to get container status \"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac\": rpc error: code = NotFound desc = could not find container \"ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac\": container with ID starting with ce4518bf1f3bbc97682ed42ed55544749ccec2d22f59d381d087c77bbaae59ac not found: ID does not exist"
Jul 09 18:38:34.959002 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.958983    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:38:34.962130 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:34.962112    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb3ff-predictor-5c99cc74cd-2x6xd"]
Jul 09 18:38:35.454462 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:35.454420    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" path="/var/lib/kubelet/pods/3ac59db0-951f-49c0-aaa0-e3937fb552ee/volumes"
Jul 09 18:38:35.812109 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:35.812072    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 18:38:39.943935 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:39.943902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:38:39.944364 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:39.944335    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:38:45.811923 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:45.811888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:38:49.945130 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:49.945089    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:38:59.944374 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:38:59.944296    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:39:09.944708 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:09.944665    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 18:39:19.945707 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:19.945677    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:39:29.422467 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:29.422439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:39:29.426875 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:29.426854    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:39:29.427587 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:29.427566    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:39:29.431931 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:39:29.431914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:44:29.451882 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:44:29.451840    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:44:29.455852 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:44:29.455830    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:44:29.456808 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:44:29.456784    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:44:29.459867 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:44:29.459847    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:47:46.240119 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:46.240083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:47:46.241621 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:46.241567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container" containerID="cri-o://f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be" gracePeriod=30
Jul 09 18:47:46.245611 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:46.242024    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kube-rbac-proxy" containerID="cri-o://b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1" gracePeriod=30
Jul 09 18:47:46.975509 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:46.975475    2561 generic.go:356] "Generic (PLEG): container finished" podID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerID="b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1" exitCode=2
Jul 09 18:47:46.975669 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:46.975528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerDied","Data":"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"}
Jul 09 18:47:49.393689 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.393666    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:47:49.496079 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.495995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config\") pod \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") "
Jul 09 18:47:49.496079 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.496070    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls\") pod \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") "
Jul 09 18:47:49.496302 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.496107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hf9b\" (UniqueName: \"kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b\") pod \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\" (UID: \"268aa6f6-f9b0-4244-80c4-0b21f3590cb3\") "
Jul 09 18:47:49.496378 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.496350    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "268aa6f6-f9b0-4244-80c4-0b21f3590cb3" (UID: "268aa6f6-f9b0-4244-80c4-0b21f3590cb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:47:49.498101 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.498070    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b" (OuterVolumeSpecName: "kube-api-access-5hf9b") pod "268aa6f6-f9b0-4244-80c4-0b21f3590cb3" (UID: "268aa6f6-f9b0-4244-80c4-0b21f3590cb3"). InnerVolumeSpecName "kube-api-access-5hf9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:47:49.498101 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.498095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "268aa6f6-f9b0-4244-80c4-0b21f3590cb3" (UID: "268aa6f6-f9b0-4244-80c4-0b21f3590cb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:47:49.596675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.596639    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:47:49.596675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.596667    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:47:49.596675 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.596678    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hf9b\" (UniqueName: \"kubernetes.io/projected/268aa6f6-f9b0-4244-80c4-0b21f3590cb3-kube-api-access-5hf9b\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:47:49.987441 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.987405    2561 generic.go:356] "Generic (PLEG): container finished" podID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerID="f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be" exitCode=0
Jul 09 18:47:49.987614 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.987464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"
Jul 09 18:47:49.987614 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.987491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerDied","Data":"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"}
Jul 09 18:47:49.987614 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.987559    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc" event={"ID":"268aa6f6-f9b0-4244-80c4-0b21f3590cb3","Type":"ContainerDied","Data":"886a1503b1f6c2c2b96da1fe2e7e559d8e051f49cee35cdeae8d56c89cf739b1"}
Jul 09 18:47:49.987614 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.987597    2561 scope.go:117] "RemoveContainer" containerID="b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"
Jul 09 18:47:49.996952 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:49.996934    2561 scope.go:117] "RemoveContainer" containerID="f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"
Jul 09 18:47:50.005136 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.005110    2561 scope.go:117] "RemoveContainer" containerID="b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"
Jul 09 18:47:50.005383 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:47:50.005364    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1\": container with ID starting with b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1 not found: ID does not exist" containerID="b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"
Jul 09 18:47:50.005439 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.005403    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1"} err="failed to get container status \"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1\": rpc error: code = NotFound desc = could not find container \"b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1\": container with ID starting with b5aaaaa531647927927facebf9165e2a488f059f90747a625954bad0fa5f83c1 not found: ID does not exist"
Jul 09 18:47:50.005439 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.005421    2561 scope.go:117] "RemoveContainer" containerID="f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"
Jul 09 18:47:50.005663 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:47:50.005643    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be\": container with ID starting with f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be not found: ID does not exist" containerID="f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"
Jul 09 18:47:50.005704 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.005670    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be"} err="failed to get container status \"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be\": rpc error: code = NotFound desc = could not find container \"f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be\": container with ID starting with f1f70543ab56aef8e8bedef800f4325de8f87d66c9019aee0b3d4a16f42732be not found: ID does not exist"
Jul 09 18:47:50.008126 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.008106    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:47:50.014057 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:50.014039    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-865f9-predictor-6f895b7954-9lbzc"]
Jul 09 18:47:51.454023 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:47:51.453953    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" path="/var/lib/kubelet/pods/268aa6f6-f9b0-4244-80c4-0b21f3590cb3/volumes"
Jul 09 18:49:29.479926 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:49:29.479824    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:49:29.484169 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:49:29.483509    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:49:29.484169 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:49:29.484131    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:49:29.487563 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:49:29.487546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:54:29.501814 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:54:29.501685    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:54:29.505980 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:54:29.505954    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:54:29.506789 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:54:29.506712    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:54:29.511069 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:54:29.511049    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:55:16.533442 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:16.533397    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:55:16.533990 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:16.533666    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container" containerID="cri-o://31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5" gracePeriod=30
Jul 09 18:55:16.533990 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:16.533732    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kube-rbac-proxy" containerID="cri-o://870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617" gracePeriod=30
Jul 09 18:55:17.601816 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:17.601784    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerID="870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617" exitCode=2
Jul 09 18:55:17.602197 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:17.601841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerDied","Data":"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"}
Jul 09 18:55:19.579838 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.579810    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:55:19.611063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.611023    2561 generic.go:356] "Generic (PLEG): container finished" podID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerID="31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5" exitCode=0
Jul 09 18:55:19.611063 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.611064    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"
Jul 09 18:55:19.611297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.611098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerDied","Data":"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"}
Jul 09 18:55:19.611297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.611164    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67" event={"ID":"6f082334-acba-45d2-885e-ba1fe7db0a74","Type":"ContainerDied","Data":"41538fd887fbaf4bd2a26be00bc158a2b62b69f9b62f7b2d9da20e8b7d883473"}
Jul 09 18:55:19.611297 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.611203    2561 scope.go:117] "RemoveContainer" containerID="870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"
Jul 09 18:55:19.618967 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.618948    2561 scope.go:117] "RemoveContainer" containerID="31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"
Jul 09 18:55:19.625870 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.625839    2561 scope.go:117] "RemoveContainer" containerID="870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"
Jul 09 18:55:19.626141 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:55:19.626124    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617\": container with ID starting with 870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617 not found: ID does not exist" containerID="870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"
Jul 09 18:55:19.626189 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.626151    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617"} err="failed to get container status \"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617\": rpc error: code = NotFound desc = could not find container \"870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617\": container with ID starting with 870d35b2c54300a2bf3ab7cf0ae306a245b4347818d3dcee16959eca3e243617 not found: ID does not exist"
Jul 09 18:55:19.626189 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.626170    2561 scope.go:117] "RemoveContainer" containerID="31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"
Jul 09 18:55:19.626417 ip-10-0-138-20 kubenswrapper[2561]: E0709 18:55:19.626392    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5\": container with ID starting with 31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5 not found: ID does not exist" containerID="31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"
Jul 09 18:55:19.626458 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.626423    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5"} err="failed to get container status \"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5\": rpc error: code = NotFound desc = could not find container \"31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5\": container with ID starting with 31525d8b6e4e9fb6169fe748e6a7690d7366ec672e4ff1bb213538862b3d30e5 not found: ID does not exist"
Jul 09 18:55:19.728622 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.728534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config\") pod \"6f082334-acba-45d2-885e-ba1fe7db0a74\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") "
Jul 09 18:55:19.728622 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.728579    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4gm2\" (UniqueName: \"kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2\") pod \"6f082334-acba-45d2-885e-ba1fe7db0a74\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") "
Jul 09 18:55:19.728622 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.728620    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls\") pod \"6f082334-acba-45d2-885e-ba1fe7db0a74\" (UID: \"6f082334-acba-45d2-885e-ba1fe7db0a74\") "
Jul 09 18:55:19.728954 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.728930    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f082334-acba-45d2-885e-ba1fe7db0a74" (UID: "6f082334-acba-45d2-885e-ba1fe7db0a74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:19.730661 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.730636    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f082334-acba-45d2-885e-ba1fe7db0a74" (UID: "6f082334-acba-45d2-885e-ba1fe7db0a74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:19.730826 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.730795    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2" (OuterVolumeSpecName: "kube-api-access-l4gm2") pod "6f082334-acba-45d2-885e-ba1fe7db0a74" (UID: "6f082334-acba-45d2-885e-ba1fe7db0a74"). InnerVolumeSpecName "kube-api-access-l4gm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:19.829456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.829413    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f082334-acba-45d2-885e-ba1fe7db0a74-proxy-tls\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:55:19.829456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.829452    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:55:19.829456 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.829463    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4gm2\" (UniqueName: \"kubernetes.io/projected/6f082334-acba-45d2-885e-ba1fe7db0a74-kube-api-access-l4gm2\") on node \"ip-10-0-138-20.ec2.internal\" DevicePath \"\""
Jul 09 18:55:19.934436 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.934408    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:55:19.938537 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:19.938513    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bed12-predictor-646d855c94-8pr67"]
Jul 09 18:55:21.454157 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:21.454081    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" path="/var/lib/kubelet/pods/6f082334-acba-45d2-885e-ba1fe7db0a74/volumes"
Jul 09 18:55:47.604068 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604033    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8wgtz/must-gather-zrb5w"]
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604292    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604301    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604313    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604318    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604334    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604342    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604347    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604356    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604361    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604369    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604374    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604415    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604422    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kube-rbac-proxy"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604429    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604435    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac59db0-951f-49c0-aaa0-e3937fb552ee" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604441    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f082334-acba-45d2-885e-ba1fe7db0a74" containerName="kserve-container"
Jul 09 18:55:47.604496 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.604446    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="268aa6f6-f9b0-4244-80c4-0b21f3590cb3" containerName="kube-rbac-proxy"
Jul 09 18:55:47.607235 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.607218    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.609823 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.609797    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8wgtz\"/\"openshift-service-ca.crt\""
Jul 09 18:55:47.609948 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.609909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8wgtz\"/\"default-dockercfg-k5z4r\""
Jul 09 18:55:47.610608 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.610595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8wgtz\"/\"kube-root-ca.crt\""
Jul 09 18:55:47.616846 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.616829    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/must-gather-zrb5w"]
Jul 09 18:55:47.650222 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.650192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9bwp\" (UniqueName: \"kubernetes.io/projected/51cd6536-a4d5-4de7-b0bd-77254b5b771d-kube-api-access-k9bwp\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.650222 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.650227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd6536-a4d5-4de7-b0bd-77254b5b771d-must-gather-output\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.751660 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.751625    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9bwp\" (UniqueName: \"kubernetes.io/projected/51cd6536-a4d5-4de7-b0bd-77254b5b771d-kube-api-access-k9bwp\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.751857 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.751673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd6536-a4d5-4de7-b0bd-77254b5b771d-must-gather-output\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.752007 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.751990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51cd6536-a4d5-4de7-b0bd-77254b5b771d-must-gather-output\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.760233 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.760213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9bwp\" (UniqueName: \"kubernetes.io/projected/51cd6536-a4d5-4de7-b0bd-77254b5b771d-kube-api-access-k9bwp\") pod \"must-gather-zrb5w\" (UID: \"51cd6536-a4d5-4de7-b0bd-77254b5b771d\") " pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:47.916863 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:47.916787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/must-gather-zrb5w"
Jul 09 18:55:48.047184 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:48.047151    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/must-gather-zrb5w"]
Jul 09 18:55:48.050018 ip-10-0-138-20 kubenswrapper[2561]: W0709 18:55:48.049990    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51cd6536_a4d5_4de7_b0bd_77254b5b771d.slice/crio-f0816f60be5f0a50ae61d2009c19d55f273284debd337fe1d3a0bc1d6ae8bc28 WatchSource:0}: Error finding container f0816f60be5f0a50ae61d2009c19d55f273284debd337fe1d3a0bc1d6ae8bc28: Status 404 returned error can't find the container with id f0816f60be5f0a50ae61d2009c19d55f273284debd337fe1d3a0bc1d6ae8bc28
Jul 09 18:55:48.051848 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:48.051831    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:55:48.717567 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:48.717534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/must-gather-zrb5w" event={"ID":"51cd6536-a4d5-4de7-b0bd-77254b5b771d","Type":"ContainerStarted","Data":"f0816f60be5f0a50ae61d2009c19d55f273284debd337fe1d3a0bc1d6ae8bc28"}
Jul 09 18:55:49.723520 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:49.723479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/must-gather-zrb5w" event={"ID":"51cd6536-a4d5-4de7-b0bd-77254b5b771d","Type":"ContainerStarted","Data":"e6da125396740253e3e81ddfc6527419f0e59195ab867aa83930e810fa9135ce"}
Jul 09 18:55:49.724070 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:49.724039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/must-gather-zrb5w" event={"ID":"51cd6536-a4d5-4de7-b0bd-77254b5b771d","Type":"ContainerStarted","Data":"648a6146f77d881eb08c561b37b65c9c5e20235f4d350c9dc8feaac10921edc7"}
Jul 09 18:55:49.739354 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:49.739305    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8wgtz/must-gather-zrb5w" podStartSLOduration=1.822201641 podStartE2EDuration="2.739288493s" totalImagesPullingTime="917.086852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:55:49.738072651 +0000 UTC m=+3080.839599697" watchObservedRunningTime="2026-07-09 18:55:49.739288493 +0000 UTC m=+3080.840815520"
Jul 09 18:55:50.397485 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:50.397449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zfwnv_c451f403-441f-4d69-992a-e55962af5083/global-pull-secret-syncer/0.log"
Jul 09 18:55:50.530162 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:50.530131    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xrz68_6bdc5c26-f7d4-4ae7-aead-a7c7a7934c10/konnectivity-agent/0.log"
Jul 09 18:55:50.581220 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:50.581194    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-20.ec2.internal_ada18c87c87425a91187e5c2f3745abe/haproxy/0.log"
Jul 09 18:55:54.947956 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:54.947928    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-rbqb5_2d300504-58da-4c86-ab1d-47d7376f0abb/monitoring-plugin/0.log"
Jul 09 18:55:55.199364 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.199277    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/node-exporter/0.log"
Jul 09 18:55:55.220536 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.220504    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/kube-rbac-proxy/0.log"
Jul 09 18:55:55.248863 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.248836    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcswq_5a7f5f9e-d482-483f-b60f-03a8f42da0ff/init-textfile/0.log"
Jul 09 18:55:55.369243 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.369179    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/prometheus/0.log"
Jul 09 18:55:55.388055 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.388024    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/config-reloader/0.log"
Jul 09 18:55:55.410934 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.410898    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/thanos-sidecar/0.log"
Jul 09 18:55:55.431494 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.431467    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/kube-rbac-proxy-web/0.log"
Jul 09 18:55:55.456325 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.456243    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/kube-rbac-proxy/0.log"
Jul 09 18:55:55.476188 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.476160    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/kube-rbac-proxy-thanos/0.log"
Jul 09 18:55:55.499004 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:55.498979    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32f7139b-924f-43fe-ac3b-d696cde592d9/init-config-reloader/0.log"
Jul 09 18:55:57.581916 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:57.581877    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/2.log"
Jul 09 18:55:57.589922 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:57.589889    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jxn6g_00a0f1dd-da67-4730-9820-8fc0ec8ac69e/console-operator/3.log"
Jul 09 18:55:58.700124 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.700083    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"]
Jul 09 18:55:58.705141 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.705117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.712317 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.712288    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"]
Jul 09 18:55:58.760324 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.760288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-proc\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.760324 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.760339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-podres\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.760587 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.760412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98c9\" (UniqueName: \"kubernetes.io/projected/ab04e36c-c282-4744-9489-0f4e6ee9032f-kube-api-access-r98c9\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.760587 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.760437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-sys\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.760587 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.760479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-lib-modules\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861524 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98c9\" (UniqueName: \"kubernetes.io/projected/ab04e36c-c282-4744-9489-0f4e6ee9032f-kube-api-access-r98c9\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861524 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-sys\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-lib-modules\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-proc\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861785 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-podres\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-sys\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-proc\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-podres\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.861951 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.861927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab04e36c-c282-4744-9489-0f4e6ee9032f-lib-modules\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:58.870224 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:58.870199    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98c9\" (UniqueName: \"kubernetes.io/projected/ab04e36c-c282-4744-9489-0f4e6ee9032f-kube-api-access-r98c9\") pod \"perf-node-gather-daemonset-lnjl4\" (UID: \"ab04e36c-c282-4744-9489-0f4e6ee9032f\") " pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:59.018167 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.017796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:59.192639 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.192559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"]
Jul 09 18:55:59.521320 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.521281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mkq2_d56b0b17-5d17-4327-87d0-f5904cb3aa04/dns/0.log"
Jul 09 18:55:59.542191 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.542165    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mkq2_d56b0b17-5d17-4327-87d0-f5904cb3aa04/kube-rbac-proxy/0.log"
Jul 09 18:55:59.655090 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.655019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vk8gc_c2afda61-2c05-44c1-a4ff-c73ef0b46e2d/dns-node-resolver/0.log"
Jul 09 18:55:59.779287 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.779236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4" event={"ID":"ab04e36c-c282-4744-9489-0f4e6ee9032f","Type":"ContainerStarted","Data":"f313632ddf76d39c63eea589e747516c405694cdfc2dd61e20e60851bbd5357f"}
Jul 09 18:55:59.779749 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.779353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4" event={"ID":"ab04e36c-c282-4744-9489-0f4e6ee9032f","Type":"ContainerStarted","Data":"ebcdf6aa81f62631b0a0e450893f3c9e12935916dbdb50b14afc87a9227fab68"}
Jul 09 18:55:59.781313 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.781281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:55:59.796705 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:55:59.796649    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4" podStartSLOduration=1.7966299000000001 podStartE2EDuration="1.7966299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:55:59.795861811 +0000 UTC m=+3090.897388838" watchObservedRunningTime="2026-07-09 18:55:59.7966299 +0000 UTC m=+3090.898156928"
Jul 09 18:56:00.158155 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:00.158124    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cc8ccb7bd-kdptw_7831265e-7320-419f-ac58-048fa4d5b8b6/registry/0.log"
Jul 09 18:56:00.231893 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:00.231859    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szr88_87db40f4-a668-401e-9eb0-737aba2f6a03/node-ca/0.log"
Jul 09 18:56:01.014141 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:01.014114    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f565474d4-kwrxx_ac28916c-868d-41f0-bfbf-b1ec473d8903/router/0.log"
Jul 09 18:56:01.371397 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:01.371371    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5rmsk_648fd878-2053-4126-b5b7-6bad43fe0141/serve-healthcheck-canary/0.log"
Jul 09 18:56:01.847662 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:01.847631    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqprt_44a23042-fd7e-4bce-8ba9-d760919382ec/kube-rbac-proxy/0.log"
Jul 09 18:56:01.869948 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:01.869916    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqprt_44a23042-fd7e-4bce-8ba9-d760919382ec/exporter/0.log"
Jul 09 18:56:01.891282 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:01.891256    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nqprt_44a23042-fd7e-4bce-8ba9-d760919382ec/extractor/0.log"
Jul 09 18:56:06.799307 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:06.799277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8wgtz/perf-node-gather-daemonset-lnjl4"
Jul 09 18:56:09.570088 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.569976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4p5sn_181c05ea-94cb-43d5-8250-0b16e9765f70/kube-multus/0.log"
Jul 09 18:56:09.642099 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.642075    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:56:09.665922 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.665871    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/egress-router-binary-copy/0.log"
Jul 09 18:56:09.696606 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.696577    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/cni-plugins/0.log"
Jul 09 18:56:09.721403 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.721359    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/bond-cni-plugin/0.log"
Jul 09 18:56:09.757238 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.757210    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/routeoverride-cni/0.log"
Jul 09 18:56:09.783413 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.783381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/whereabouts-cni-bincopy/0.log"
Jul 09 18:56:09.807831 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:09.807808    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k7vpg_5149ade1-96ea-4336-90a2-3b0f53c2e8da/whereabouts-cni/0.log"
Jul 09 18:56:10.358934 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:10.358905    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-blwkj_c5f59614-c402-4162-87a3-ab09262ca067/network-metrics-daemon/0.log"
Jul 09 18:56:10.378020 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:10.377984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-blwkj_c5f59614-c402-4162-87a3-ab09262ca067/kube-rbac-proxy/0.log"
Jul 09 18:56:11.314292 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.314252    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-controller/0.log"
Jul 09 18:56:11.338571 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.338510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/1.log"
Jul 09 18:56:11.338788 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.338597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovn-acl-logging/0.log"
Jul 09 18:56:11.362661 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.362632    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/kube-rbac-proxy-node/0.log"
Jul 09 18:56:11.388137 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.388108    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:56:11.406211 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.406188    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/northd/0.log"
Jul 09 18:56:11.426900 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.426870    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/nbdb/0.log"
Jul 09 18:56:11.446326 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.446301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/sbdb/0.log"
Jul 09 18:56:11.506937 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:11.506909    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5t7cr_fb87d44a-492c-434b-8dd9-23798314b545/ovnkube-controller/0.log"
Jul 09 18:56:12.899271 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:12.899241    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-qknbm_59c8f550-78fc-4f3d-aede-39473570c0cd/check-endpoints/0.log"
Jul 09 18:56:12.923035 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:12.923009    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2dwcb_8b2b359c-3de6-404d-b061-b0fe458d8a4a/network-check-target-container/0.log"
Jul 09 18:56:13.929735 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:13.929693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8cgn4_79d1e3ce-e800-4992-a055-50acfe377bf3/iptables-alerter/0.log"
Jul 09 18:56:14.692491 ip-10-0-138-20 kubenswrapper[2561]: I0709 18:56:14.692462    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zjs8g_5e973edd-61b9-4a2d-b95d-31f6bdb249ab/tuned/0.log"