Jul 06 19:09:41.897764 ip-10-0-143-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 19:09:41.897776 ip-10-0-143-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 19:09:41.897783 ip-10-0-143-63 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 19:09:41.898002 ip-10-0-143-63 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 19:09:52.960152 ip-10-0-143-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 19:09:52.960170 ip-10-0-143-63 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a57c5da3dc4d4c8abe623ac87c79110c -- Jul 06 19:11:28.359636 ip-10-0-143-63 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:11:28.789021 ip-10-0-143-63 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:28.789021 ip-10-0-143-63 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:11:28.789021 ip-10-0-143-63 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:28.789021 ip-10-0-143-63 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:11:28.789021 ip-10-0-143-63 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:11:28.792023 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.791935 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:11:28.796024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796007 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.796024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796024 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796028 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796032 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796036 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796038 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796041 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796044 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796046 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796049 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796051 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796054 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796057 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796059 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796063 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796065 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796068 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796071 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796073 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796076 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796078 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.796087 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796081 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796084 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796087 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796089 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796092 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796095 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796098 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796100 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796103 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796105 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796108 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796111 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796113 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796117 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796121 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796124 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796127 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796131 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796133 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.796570 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796136 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796138 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796141 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796143 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796146 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796150 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796153 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796156 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796159 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796163 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796166 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796169 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796172 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796174 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796176 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796180 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796183 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796185 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796188 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.797049 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796190 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796193 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796196 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796198 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796201 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796203 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796205 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796208 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796211 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796213 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796215 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796218 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796221 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796223 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796226 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796228 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796231 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796233 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796235 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796239 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.797516 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796242 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796244 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796246 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796249 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796251 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796253 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796256 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796258 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796262 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796265 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796267 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796270 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796272 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796275 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796277 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796280 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796282 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796285 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796287 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.796289 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.798005 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797907 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797914 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797918 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797921 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797924 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797927 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797931 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797933 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797937 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797940 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797943 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797946 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797949 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797952 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797955 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797958 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797962 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797966 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797969 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797972 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.798583 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797977 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797980 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797983 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797985 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797988 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797990 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797993 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797995 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.797997 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798000 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798002 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798005 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798008 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798011 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798013 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798016 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798018 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798021 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798024 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798027 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.799086 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798030 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798032 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798037 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798040 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798042 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798045 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798048 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798050 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798053 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798055 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798057 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798060 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798063 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798066 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798068 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798072 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798076 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798078 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798081 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.799606 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798084 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798087 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798089 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798092 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798094 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798096 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798099 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798101 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798104 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798106 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798109 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798112 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798115 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798118 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798120 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798123 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798125 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798128 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798130 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798133 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.800068 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798135 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798138 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798140 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798142 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798145 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798150 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798152 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798155 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798157 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798160 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798163 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798165 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798168 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798170 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798173 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798175 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798178 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798180 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798182 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798185 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.800552 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798582 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798587 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798591 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798594 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798598 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798602 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798604 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798607 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798610 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798612 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798615 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798617 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798620 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798622 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798625 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798627 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798630 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798633 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798636 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798638 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.801053 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798641 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798643 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798646 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798649 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798651 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798654 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798657 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798660 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798662 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798665 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798667 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798669 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798672 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798674 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798676 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798679 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798681 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798692 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798695 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798697 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.801546 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798700 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798702 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798705 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798707 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798709 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798712 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798714 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798717 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798719 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798722 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798724 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798727 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798729 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798732 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798734 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798737 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798739 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798742 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798745 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798747 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.802021 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798750 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798752 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798755 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798757 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798761 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798763 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798765 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798768 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798771 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798773 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798776 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798778 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798781 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798784 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798787 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798790 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798795 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798798 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798801 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.802535 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798805 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798808 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798811 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798814 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798817 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798819 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798821 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798824 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798826 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798829 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798831 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798834 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798837 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798839 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798841 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798843 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798846 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798848 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798851 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.798853 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.802998 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798926 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798934 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798940 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798944 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798948 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798951 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798955 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798960 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798963 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798965 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798968 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798971 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798974 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798977 2558 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798980 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798983 2558 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798986 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798989 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798994 2558 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.798997 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799000 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799003 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799006 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799010 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:11:28.803539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799012 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799015 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799018 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799021 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799024 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799027 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799030 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799033 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799037 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799039 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799042 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799045 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799048 2558 flags.go:64] FLAG: --enable-server="true" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799052 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799056 2558 flags.go:64] FLAG: --event-burst="100" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799059 2558 flags.go:64] FLAG: --event-qps="50" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799062 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799065 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799067 2558 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799071 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799074 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799076 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799081 2558 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799084 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799087 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:11:28.804135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799090 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799093 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799095 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799098 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799101 2558 flags.go:64] FLAG: --feature-gates="" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799105 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799108 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799119 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799123 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799125 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799129 2558 flags.go:64] FLAG: --help="false" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799132 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-143-63.ec2.internal" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799135 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799137 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799140 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799143 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799147 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799150 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799152 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799155 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799158 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799161 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799164 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799167 2558 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:11:28.804754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799169 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799172 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799175 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799177 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799180 2558 flags.go:64] FLAG: --lock-file="" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799184 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799188 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799192 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799197 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799200 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799203 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799205 2558 flags.go:64] FLAG: --logging-format="text" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799221 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799225 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799228 2558 flags.go:64] FLAG: --manifest-url="" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799231 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799235 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799238 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799242 2558 flags.go:64] FLAG: --max-pods="110" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799245 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799248 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799250 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799254 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799257 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799259 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:11:28.805334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799262 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799269 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799272 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799275 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799278 2558 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799281 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799287 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799289 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799293 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799295 2558 flags.go:64] FLAG: --port="10250" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799298 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799301 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034dfc36108ed5f4f" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799304 2558 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799309 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799312 2558 flags.go:64] FLAG: --register-node="true" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799315 2558 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799319 2558 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799321 2558 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799324 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799327 2558 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799331 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799334 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799337 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799339 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799342 2558 flags.go:64] FLAG: --runonce="false" Jul 06 19:11:28.805969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799344 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799364 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799367 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799370 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799373 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799376 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799379 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799382 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799385 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799388 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799390 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799393 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799396 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799399 2558 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799402 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799408 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799410 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799413 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799417 2558 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799420 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799425 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799428 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799431 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799433 2558 flags.go:64] FLAG: --v="2" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799437 2558 flags.go:64] FLAG: --version="false" Jul 06 19:11:28.806598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799444 2558 flags.go:64] FLAG: --vmodule="" Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799449 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799452 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799566 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799570 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799573 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799576 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799578 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799581 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799583 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799586 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799589 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799591 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799594 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799596 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799599 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799601 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799604 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799606 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799609 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.807234 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799611 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799615 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799618 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799620 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799623 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799625 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799628 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799633 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799636 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799638 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799641 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799644 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799646 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799649 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799651 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799654 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799656 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799659 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799661 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799664 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.807747 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799666 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799668 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799671 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799674 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799679 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799682 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799686 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799689 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799692 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799695 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799698 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799701 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799704 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799706 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799712 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799714 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799717 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799719 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799722 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.808225 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799726 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799728 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799731 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799733 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799736 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799739 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799741 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799744 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799746 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799749 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799751 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799754 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799756 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799759 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799761 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799764 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799766 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799769 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799771 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799774 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.808698 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799776 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799779 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799781 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799783 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799786 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799789 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799791 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799793 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799796 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799799 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799801 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799803 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799809 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799811 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799813 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799816 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799819 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799821 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799823 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799827 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.809178 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799830 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799833 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.799836 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.799841 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.806944 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.806960 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.807835 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.807842 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807900 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807905 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807910 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807914 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807917 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807920 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.809684 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807923 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807926 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807930 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807933 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807935 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807938 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807944 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807947 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807950 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807953 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807955 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807958 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807961 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807963 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807965 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807968 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807971 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807973 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807976 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807978 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.810024 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807981 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807983 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807986 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807989 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807992 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807994 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807997 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.807999 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808002 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808004 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808007 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808009 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808011 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808014 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808016 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808019 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808021 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808023 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808026 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808029 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.810515 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808031 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808033 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808036 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808038 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808041 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808043 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808046 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808048 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808050 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808053 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808055 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808058 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808061 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808063 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808067 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808069 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808072 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808074 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808078 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808081 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.810998 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808084 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808087 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808091 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808093 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808095 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808098 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808100 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808102 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808105 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808107 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808110 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808112 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808115 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808118 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808120 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808123 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808125 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808127 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808130 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808132 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.811559 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808135 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808138 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808142 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808145 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808148 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808151 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808154 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808156 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808159 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808161 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808163 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808166 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808168 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.808173 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808285 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:11:28.812027 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808290 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808292 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808295 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808298 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808301 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808304 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808306 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808309 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808311 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808314 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808317 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808320 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808325 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808330 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808333 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808336 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808339 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808342 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808344 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:11:28.812404 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808359 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808362 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808365 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808367 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808370 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808373 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808375 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808378 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808380 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808383 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808385 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808388 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808390 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808393 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808395 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808397 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808400 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808402 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808405 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808407 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:11:28.812868 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808409 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808412 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808414 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808417 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808420 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808422 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808425 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808427 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808430 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808432 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808435 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808437 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808439 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808442 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808445 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808447 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808449 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808452 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808455 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808457 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:11:28.813342 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808460 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808462 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808465 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808467 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808470 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808472 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808474 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808477 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808479 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808481 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808484 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808486 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808488 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808490 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808493 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808495 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808498 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808500 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808503 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808505 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:11:28.813819 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808508 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808510 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808512 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808515 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808518 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808521 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808524 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808527 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808530 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808532 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808535 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808538 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808540 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808543 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808545 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808548 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808550 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808552 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:11:28.814290 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:28.808555 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.808560 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.808673 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.811060 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.812802 2558 server.go:1019] "Starting client certificate rotation" Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.812900 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:28.814725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.812933 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:11:28.818364 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.818315 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:28.819972 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.819955 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:11:28.832361 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.832323 2558 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:11:28.834882 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.834865 2558 log.go:25] "Validated CRI v1 image API" Jul 06 19:11:28.835767 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.835752 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:11:28.838387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.838341 2558 fs.go:135] Filesystem UUIDs: map[4f41141b-6885-46e4-b774-42ddd1e8b00b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ba4ec567-b4c6-4af7-ac79-188681866344:/dev/nvme0n1p4] Jul 06 19:11:28.838433 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.838388 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:11:28.842756 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.842647 2558 manager.go:216] Machine: {Timestamp:2026-07-06 19:11:28.842149774 +0000 UTC m=+0.371494791 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082355 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca5ff192ddc7652ba4efd3e08cd2f SystemUUID:ec2ca5ff-192d-dc76-52ba-4efd3e08cd2f BootID:a57c5da3-dc4d-4c8a-be62-3ac87c79110c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:9a:23:4c:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:9a:23:4c:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:b4:fe:75:5d:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:11:28.842756 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.842749 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:11:28.842854 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.842825 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:11:28.846384 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.846338 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:11:28.846547 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.846388 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:11:28.847268 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.847256 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:11:28.847319 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.847274 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:11:28.847319 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.847286 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:28.847319 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.847302 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:11:28.849251 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.849238 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:11:28.851414 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.851398 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:28.853903 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.853889 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:28.854036 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854025 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:11:28.854088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854074 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:11:28.854143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854089 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:11:28.854143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854106 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:11:28.854143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854115 2558 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:11:28.854143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854123 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:11:28.854533 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854521 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:28.854565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.854542 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:11:28.858561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.858534 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:11:28.860970 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.860956 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:11:28.861045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.860986 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:11:28.863081 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863069 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:11:28.863128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863085 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:11:28.863128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863092 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:11:28.863128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863097 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:11:28.863128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863107 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:11:28.863128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863121 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863130 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863136 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863143 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863149 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863157 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:11:28.863257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.863163 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:11:28.864143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.864132 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:11:28.864180 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.864147 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:11:28.867591 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.867576 2558 server.go:1268] "Started kubelet" Jul 06 19:11:28.867755 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.867696 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:11:28.867815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.867782 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:11:28.868428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.868396 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:11:28.868516 ip-10-0-143-63 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:11:28.868802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.868780 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:11:28.869430 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.869333 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:11:28.869430 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.869408 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:11:28.869906 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.869888 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:11:28.871398 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.871381 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:11:28.873108 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.873086 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:11:28.874007 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.872813 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-63.ec2.internal.18bfc91982c86a14 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-63.ec2.internal,UID:ip-10-0-143-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-63.ec2.internal,},FirstTimestamp:2026-07-06 19:11:28.867543572 +0000 UTC m=+0.396888587,LastTimestamp:2026-07-06 19:11:28.867543572 +0000 UTC m=+0.396888587,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-63.ec2.internal,}" Jul 06 19:11:28.880151 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.880127 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:11:28.880151 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.880143 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:28.880287 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.880266 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:28.880960 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.880928 2558 factory.go:55] Registering systemd factory Jul 06 19:11:28.880960 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.880949 2558 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:11:28.881140 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881009 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:11:28.881140 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881023 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:11:28.881235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881186 2558 factory.go:153] Registering CRI-O factory Jul 06 19:11:28.881235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881198 2558 factory.go:223] Registration of the crio container factory successfully Jul 06 19:11:28.881313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881252 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:11:28.881313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881275 2558 factory.go:103] Registering Raw factory Jul 06 19:11:28.881313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881289 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:11:28.881313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881299 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:11:28.881313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881292 2558 manager.go:1196] Started watching for new ooms in manager Jul 06 19:11:28.881604 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.881123 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:11:28.882039 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.882021 2558 manager.go:349] Starting recovery of all containers Jul 06 19:11:28.882304 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.882282 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 19:11:28.882519 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.882482 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:11:28.892936 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.892756 2558 manager.go:354] Recovery completed Jul 06 19:11:28.895728 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.895666 2558 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:11:28.898572 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.898557 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:28.900640 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.900621 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:28.900704 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.900657 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:28.900704 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.900667 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:28.901178 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.901163 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:11:28.901178 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.901177 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:11:28.901262 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.901190 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:28.901262 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.901196 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:11:28.902489 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.902472 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:28.902542 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.902500 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:28.902542 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.902509 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:28.903316 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.903304 2558 policy_none.go:49] "None policy: Start" Jul 06 19:11:28.903372 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.903319 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:11:28.903372 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.903329 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:11:28.905206 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.905196 2558 policy_none.go:47] "Start" Jul 06 19:11:28.912144 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.912101 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l5mln" Jul 06 19:11:28.915690 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.915614 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-63.ec2.internal.18bfc91984c17055 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-63.ec2.internal,UID:ip-10-0-143-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-63.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-63.ec2.internal,},FirstTimestamp:2026-07-06 19:11:28.900640853 +0000 UTC m=+0.429985872,LastTimestamp:2026-07-06 19:11:28.900640853 +0000 UTC m=+0.429985872,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-63.ec2.internal,}" Jul 06 19:11:28.921747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.921729 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l5mln" Jul 06 19:11:28.923267 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.923204 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-63.ec2.internal.18bfc91984c1c2cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-63.ec2.internal,UID:ip-10-0-143-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-63.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-63.ec2.internal,},FirstTimestamp:2026-07-06 19:11:28.900661965 +0000 UTC m=+0.430006982,LastTimestamp:2026-07-06 19:11:28.900661965 +0000 UTC m=+0.430006982,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-63.ec2.internal,}" Jul 06 19:11:28.955978 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.955961 2558 manager.go:337] "Starting Device Plugin manager" Jul 06 19:11:28.956091 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.955996 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:11:28.956091 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956009 2558 server.go:87] "Starting device plugin registration server" Jul 06 19:11:28.956700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956234 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:11:28.956700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956260 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:11:28.956700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956500 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:11:28.956700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956565 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:11:28.956700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:28.956572 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:11:28.957115 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.957098 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:11:28.957220 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:28.957165 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.009694 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.009660 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:11:29.011832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.011047 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:11:29.011832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.011088 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:11:29.011832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.011118 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:11:29.011832 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.011163 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:11:29.013964 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.013941 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:29.057462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.057390 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.058235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.058215 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.058327 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.058251 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.058327 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.058264 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:29.058327 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.058291 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.067991 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.067973 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.068076 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.068002 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-63.ec2.internal\": node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.104287 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.104255 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.112251 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.112227 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal"] Jul 06 19:11:29.112336 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.112288 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.113829 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.113811 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.113918 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.113849 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.113918 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.113873 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:29.116233 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116221 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.116343 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116329 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.116457 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116385 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.116937 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116918 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.117025 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116941 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.117025 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.116955 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:29.117025 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.117020 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.117120 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.117048 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.117120 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.117061 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:29.119189 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.119176 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.119248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.119199 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.119829 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.119816 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.119882 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.119840 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.119882 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.119854 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:29.149014 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.148988 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-63.ec2.internal\" not found" node="ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.153304 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.153289 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-63.ec2.internal\" not found" node="ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.182490 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.182463 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.182591 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.182495 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.182591 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.182515 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3312f4e0b6092b33b85bb20010305980-config\") pod \"kube-apiserver-proxy-ip-10-0-143-63.ec2.internal\" (UID: \"3312f4e0b6092b33b85bb20010305980\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.205344 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.205318 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.283153 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283122 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.283256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283158 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.283256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283176 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3312f4e0b6092b33b85bb20010305980-config\") pod \"kube-apiserver-proxy-ip-10-0-143-63.ec2.internal\" (UID: \"3312f4e0b6092b33b85bb20010305980\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.283256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283223 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.283256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283248 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3312f4e0b6092b33b85bb20010305980-config\") pod \"kube-apiserver-proxy-ip-10-0-143-63.ec2.internal\" (UID: \"3312f4e0b6092b33b85bb20010305980\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.283420 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.283219 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61e1073d4c048b344be043b81c15583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal\" (UID: \"61e1073d4c048b344be043b81c15583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.305853 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.305828 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.406682 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.406625 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.451821 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.451787 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.456856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.456681 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:29.507239 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.507202 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.607741 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.607708 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.707508 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.707443 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:29.708405 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.708389 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.797546 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.797509 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:29.809150 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.809125 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.812280 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.812265 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:11:29.812427 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.812409 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:11:29.812484 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.812409 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:11:29.812484 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.812427 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:11:29.910051 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:29.910015 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:29.927150 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.927101 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:06:29 +0000 UTC" deadline="2027-12-12 08:33:07.833135735 +0000 UTC" Jul 06 19:11:29.927150 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.927142 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12565h21m37.905999386s" Jul 06 19:11:29.949287 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.949264 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:11:29.950474 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.950455 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:11:29.950581 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.950486 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:11:29.950581 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:29.950496 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:11:30.011148 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:30.011077 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-63.ec2.internal\" not found" Jul 06 19:11:30.091156 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.091128 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:30.161761 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:30.161714 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e1073d4c048b344be043b81c15583d.slice/crio-07dd01309208980a344f50f4cfffc02e3c31bb827dfd040c6159bbcbc52b9512 WatchSource:0}: Error finding container 07dd01309208980a344f50f4cfffc02e3c31bb827dfd040c6159bbcbc52b9512: Status 404 returned error can't find the container with id 07dd01309208980a344f50f4cfffc02e3c31bb827dfd040c6159bbcbc52b9512 Jul 06 19:11:30.162141 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:30.162121 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3312f4e0b6092b33b85bb20010305980.slice/crio-f843448d6c46145a3f0bd0a5518a1a2971bdcec809ab48fb60901db3cdc03089 WatchSource:0}: Error finding container f843448d6c46145a3f0bd0a5518a1a2971bdcec809ab48fb60901db3cdc03089: Status 404 returned error can't find the container with id f843448d6c46145a3f0bd0a5518a1a2971bdcec809ab48fb60901db3cdc03089 Jul 06 19:11:30.168290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.168275 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:11:30.181114 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.181091 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" Jul 06 19:11:30.197760 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.197729 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:30.198710 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.198697 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" Jul 06 19:11:30.213451 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.213435 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:11:30.855682 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.855590 2558 apiserver.go:52] "Watching apiserver" Jul 06 19:11:30.862610 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.862413 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:11:30.864012 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.863981 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k","openshift-dns/node-resolver-r7jbm","openshift-image-registry/node-ca-5qbrh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal","openshift-multus/network-metrics-daemon-q7p7g","openshift-ovn-kubernetes/ovnkube-node-gzcfs","kube-system/konnectivity-agent-ghvvr","openshift-cluster-node-tuning-operator/tuned-85fsk","openshift-multus/multus-additional-cni-plugins-4drnz","openshift-multus/multus-jm68t","openshift-network-diagnostics/network-check-target-22lf6","openshift-network-operator/iptables-alerter-t9rlb","kube-system/global-pull-secret-syncer-x8h8p","kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal"] Jul 06 19:11:30.867715 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.867114 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.869846 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.869525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:30.869846 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.869699 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.870093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.870063 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:11:30.870093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.870072 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-56pnx\"" Jul 06 19:11:30.870264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.870112 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:11:30.870264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.870161 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.874308 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.874277 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:11:30.874559 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.874542 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:11:30.874606 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.874587 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cxhq2\"" Jul 06 19:11:30.874814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.874795 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:30.875568 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.875549 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.876758 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.876737 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.877544 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.877524 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9nznv\"" Jul 06 19:11:30.877957 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.877935 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:11:30.878110 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.877855 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.878216 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.877777 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.878305 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.878288 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:30.878401 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:30.878376 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:30.878474 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.878390 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fxlq4\"" Jul 06 19:11:30.878474 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.878451 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.880003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.879893 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:30.881465 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.881424 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:11:30.882762 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.882744 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:30.884941 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.884921 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:30.885036 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:30.884984 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:30.885198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.885173 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.885297 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.885285 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvgzt\"" Jul 06 19:11:30.885381 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.885316 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.887426 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.887406 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:30.890468 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.890392 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:30.890618 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.890560 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6vfsn\"" Jul 06 19:11:30.890685 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.890629 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:11:30.890757 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.890692 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:11:30.892801 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.892767 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:30.893187 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893164 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysconfig\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893202 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-kubernetes\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893234 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-multus\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.893337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893259 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-conf-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.893337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893304 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xx2\" (UniqueName: \"kubernetes.io/projected/35bd2938-91c2-44d2-b438-25f4c431cccd-kube-api-access-j8xx2\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.893337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893328 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4295aa48-adaf-4a8f-b821-38ff2f059993-agent-certs\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893351 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-modprobe-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893375 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-sys\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-lib-modules\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893453 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt2w\" (UniqueName: \"kubernetes.io/projected/991de766-d219-4006-a9d9-99cb2b6ac4a3-kube-api-access-bpt2w\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893458 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.893514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893457 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:11:30.893731 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.893478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-os-release\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898069 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898043 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-k8s-cni-cncf-io\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898169 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898086 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-bin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898169 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898121 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-run\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.898254 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898174 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:30.898254 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898183 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-tuned\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.898254 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898217 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xlc\" (UniqueName: \"kubernetes.io/projected/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-kube-api-access-p6xlc\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:30.898254 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898244 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-cni-binary-copy\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898276 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsdz\" (UniqueName: \"kubernetes.io/projected/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-kube-api-access-qpsdz\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:30.898412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898307 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.898412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898340 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-var-lib-kubelet\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.898412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898370 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-hostroot\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898432 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-daemon-config\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898463 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-etc-kubernetes\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.898595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898490 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:30.898741 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:30.898599 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:30.898741 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898494 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-host\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:30.898741 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-conf\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.898896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898751 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:30.898896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4g4q\" (UniqueName: \"kubernetes.io/projected/3b192ffc-504c-4231-9c5a-efa16fdac630-kube-api-access-c4g4q\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:30.898896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898851 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899040 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898893 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-multus-certs\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899040 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898904 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:11:30.899040 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898926 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-tmp\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.899040 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.898972 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-cnibin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899040 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899003 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b192ffc-504c-4231-9c5a-efa16fdac630-tmp-dir\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899066 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-system-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899103 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-kubelet\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899137 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b192ffc-504c-4231-9c5a-efa16fdac630-hosts-file\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899166 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4295aa48-adaf-4a8f-b821-38ff2f059993-konnectivity-ca\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899197 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-serviceca\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899245 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-systemd\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899262 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:11:30.899295 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-host\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.899775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899331 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-socket-dir-parent\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899370 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-netns\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.899775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899393 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:11:30.899775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899660 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.899966 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.899835 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8cb24\"" Jul 06 19:11:30.900147 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.900125 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vt98f\"" Jul 06 19:11:30.900232 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.900199 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:11:30.900457 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.900418 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.900832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.900789 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.903491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.902923 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:11:30.903491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.902958 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:11:30.903491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.903237 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:11:30.903491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.903290 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xxs5g\"" Jul 06 19:11:30.903491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.903390 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:11:30.929676 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.929598 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2xt8" Jul 06 19:11:30.940103 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.940012 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t2xt8" Jul 06 19:11:30.982985 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.982950 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:11:30.999770 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999729 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4295aa48-adaf-4a8f-b821-38ff2f059993-agent-certs\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999787 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-modprobe-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-lib-modules\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999849 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-os-release\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999884 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-k8s-cni-cncf-io\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:30.999976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999948 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-systemd-units\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:30.999989 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-var-lib-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-run\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000038 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-tuned\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000064 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xlc\" (UniqueName: \"kubernetes.io/projected/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-kube-api-access-p6xlc\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000090 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-cni-binary-copy\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000130 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-os-release\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-netns\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000184 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000208 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-node-log\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000246 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlz2t\" (UniqueName: \"kubernetes.io/projected/be832039-1f5e-43d8-a376-d94d8be6ae45-kube-api-access-tlz2t\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.000281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000248 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000288 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cnibin\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000313 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsdz\" (UniqueName: \"kubernetes.io/projected/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-kube-api-access-qpsdz\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-hostroot\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000429 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000457 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be832039-1f5e-43d8-a376-d94d8be6ae45-host-slash\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000483 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000510 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-host\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000535 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-conf\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4g4q\" (UniqueName: \"kubernetes.io/projected/3b192ffc-504c-4231-9c5a-efa16fdac630-kube-api-access-c4g4q\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-multus-certs\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-dbus\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-slash\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000686 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-log-socket\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000710 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-netd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000737 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-cnibin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000764 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxxh\" (UniqueName: \"kubernetes.io/projected/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-kube-api-access-hdxxh\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000790 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-etc-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000819 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75355291-7dac-483c-ac38-bf786fe91684-ovn-node-metrics-cert\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000827 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-lib-modules\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000846 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lhf\" (UniqueName: \"kubernetes.io/projected/3caf62a2-ab88-4e90-989d-46df317f1e1c-kube-api-access-x7lhf\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000884 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b192ffc-504c-4231-9c5a-efa16fdac630-tmp-dir\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000901 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-os-release\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000909 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be832039-1f5e-43d8-a376-d94d8be6ae45-iptables-alerter-script\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000945 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-k8s-cni-cncf-io\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000948 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-script-lib\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001000 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b192ffc-504c-4231-9c5a-efa16fdac630-hosts-file\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001062 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-run\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001070 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4295aa48-adaf-4a8f-b821-38ff2f059993-konnectivity-ca\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-systemd\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001131 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-host\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001166 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xx2\" (UniqueName: \"kubernetes.io/projected/35bd2938-91c2-44d2-b438-25f4c431cccd-kube-api-access-j8xx2\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.001838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001195 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-bin\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001223 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001249 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-config\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001276 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-kubernetes\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001304 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-multus\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-conf-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001359 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001384 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m5f\" (UniqueName: \"kubernetes.io/projected/75355291-7dac-483c-ac38-bf786fe91684-kube-api-access-j8m5f\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001464 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-device-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-sys\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001530 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt2w\" (UniqueName: \"kubernetes.io/projected/991de766-d219-4006-a9d9-99cb2b6ac4a3-kube-api-access-bpt2w\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001558 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-bin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001585 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001619 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-var-lib-kubelet\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001690 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-daemon-config\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.002692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001723 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-etc-kubernetes\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001750 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-env-overrides\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001775 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001874 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001893 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-hostroot\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001901 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-tmp\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001973 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-kubelet-config\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.001998 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-host\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002015 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-systemd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002041 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-ovn\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002070 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-system-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002099 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-kubelet\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002128 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-conf\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002381 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-multus-certs\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.003671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002502 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b192ffc-504c-4231-9c5a-efa16fdac630-tmp-dir\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002527 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-cnibin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.000735 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-modprobe-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002127 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002604 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b192ffc-504c-4231-9c5a-efa16fdac630-hosts-file\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002611 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-kubelet\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002648 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-serviceca\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002677 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-socket-dir-parent\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002703 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-netns\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002730 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-system-cni-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002759 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysconfig\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.002849 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysconfig\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003166 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4295aa48-adaf-4a8f-b821-38ff2f059993-konnectivity-ca\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003259 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-systemd\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003300 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-serviceca\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003309 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-host\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003384 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-socket-dir-parent\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003449 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-run-netns\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.004526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003666 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-sysctl-d\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003754 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-kubernetes\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003803 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-multus\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003847 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-conf-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.003941 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-sys\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.004175 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-cni-bin\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.004313 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/991de766-d219-4006-a9d9-99cb2b6ac4a3-var-lib-kubelet\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.004882 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-daemon-config\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.004949 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-etc-kubernetes\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.005059 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.005124 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:31.505103176 +0000 UTC m=+3.034448197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.005199 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4295aa48-adaf-4a8f-b821-38ff2f059993-agent-certs\") pod \"konnectivity-agent-ghvvr\" (UID: \"4295aa48-adaf-4a8f-b821-38ff2f059993\") " pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:31.005462 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.005382 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-multus-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.006053 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.005536 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-host-var-lib-kubelet\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.006053 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.005612 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35bd2938-91c2-44d2-b438-25f4c431cccd-system-cni-dir\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.006053 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.005732 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35bd2938-91c2-44d2-b438-25f4c431cccd-cni-binary-copy\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.007260 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.007238 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-tmp\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.009702 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.009652 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/991de766-d219-4006-a9d9-99cb2b6ac4a3-etc-tuned\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.013177 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.013147 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4g4q\" (UniqueName: \"kubernetes.io/projected/3b192ffc-504c-4231-9c5a-efa16fdac630-kube-api-access-c4g4q\") pod \"node-resolver-r7jbm\" (UID: \"3b192ffc-504c-4231-9c5a-efa16fdac630\") " pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.015065 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.014873 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xx2\" (UniqueName: \"kubernetes.io/projected/35bd2938-91c2-44d2-b438-25f4c431cccd-kube-api-access-j8xx2\") pod \"multus-jm68t\" (UID: \"35bd2938-91c2-44d2-b438-25f4c431cccd\") " pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.015065 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.015017 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsdz\" (UniqueName: \"kubernetes.io/projected/3e1ddb84-c2f2-42ce-9438-e2b4bf546381-kube-api-access-qpsdz\") pod \"node-ca-5qbrh\" (UID: \"3e1ddb84-c2f2-42ce-9438-e2b4bf546381\") " pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.015065 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.015017 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt2w\" (UniqueName: \"kubernetes.io/projected/991de766-d219-4006-a9d9-99cb2b6ac4a3-kube-api-access-bpt2w\") pod \"tuned-85fsk\" (UID: \"991de766-d219-4006-a9d9-99cb2b6ac4a3\") " pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.017200 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.017147 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" event={"ID":"3312f4e0b6092b33b85bb20010305980","Type":"ContainerStarted","Data":"f843448d6c46145a3f0bd0a5518a1a2971bdcec809ab48fb60901db3cdc03089"} Jul 06 19:11:31.018339 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.018291 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xlc\" (UniqueName: \"kubernetes.io/projected/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-kube-api-access-p6xlc\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:31.018687 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.018660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" event={"ID":"61e1073d4c048b344be043b81c15583d","Type":"ContainerStarted","Data":"07dd01309208980a344f50f4cfffc02e3c31bb827dfd040c6159bbcbc52b9512"} Jul 06 19:11:31.103154 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103113 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103191 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-kubelet-config\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-systemd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-ovn\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103280 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103327 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-kubelet\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-system-cni-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-systemd-units\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-var-lib-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103474 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-os-release\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103489 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-ovn\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103504 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-netns\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103541 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-netns\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103540 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-node-log\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103622 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlz2t\" (UniqueName: \"kubernetes.io/projected/be832039-1f5e-43d8-a376-d94d8be6ae45-kube-api-access-tlz2t\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103660 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cnibin\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103694 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103820 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be832039-1f5e-43d8-a376-d94d8be6ae45-host-slash\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.103884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103875 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-dbus\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-slash\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103963 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-log-socket\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.103987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-netd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104029 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxxh\" (UniqueName: \"kubernetes.io/projected/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-kube-api-access-hdxxh\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104054 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-etc-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104078 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75355291-7dac-483c-ac38-bf786fe91684-ovn-node-metrics-cert\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104105 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lhf\" (UniqueName: \"kubernetes.io/projected/3caf62a2-ab88-4e90-989d-46df317f1e1c-kube-api-access-x7lhf\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be832039-1f5e-43d8-a376-d94d8be6ae45-iptables-alerter-script\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-script-lib\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104248 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104316 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-kubelet\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-bin\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104389 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-sys-fs\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104401 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.104561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104450 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-config\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104504 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-system-cni-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104547 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-kubelet-config\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104581 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m5f\" (UniqueName: \"kubernetes.io/projected/75355291-7dac-483c-ac38-bf786fe91684-kube-api-access-j8m5f\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104604 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-systemd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104618 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104652 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-run-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104670 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-device-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104716 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-node-log\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104761 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104792 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-env-overrides\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.104968 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-registration-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105079 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cnibin\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105135 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105198 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-systemd-units\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.105481 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105247 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-var-lib-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105310 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-os-release\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105350 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-bin\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105388 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105604 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-binary-copy\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105901 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be832039-1f5e-43d8-a376-d94d8be6ae45-host-slash\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.105987 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-config\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106047 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106205 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.106301 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.106284 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.106357 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:31.606326202 +0000 UTC m=+3.135671223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106441 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106590 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61c0fbb0-8a79-474c-a770-2c14e44043b0-dbus\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106687 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-slash\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106731 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-log-socket\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106776 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-cni-netd\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.107119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106886 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-socket-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.107119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106944 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3caf62a2-ab88-4e90-989d-46df317f1e1c-device-dir\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.107119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.106986 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-host-run-ovn-kubernetes\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.107119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.107050 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75355291-7dac-483c-ac38-bf786fe91684-etc-openvswitch\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.107559 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.107525 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-env-overrides\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.107816 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.107764 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/be832039-1f5e-43d8-a376-d94d8be6ae45-iptables-alerter-script\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.107937 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.107861 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75355291-7dac-483c-ac38-bf786fe91684-ovnkube-script-lib\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.110146 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.110097 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75355291-7dac-483c-ac38-bf786fe91684-ovn-node-metrics-cert\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.119297 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.119267 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlz2t\" (UniqueName: \"kubernetes.io/projected/be832039-1f5e-43d8-a376-d94d8be6ae45-kube-api-access-tlz2t\") pod \"iptables-alerter-t9rlb\" (UID: \"be832039-1f5e-43d8-a376-d94d8be6ae45\") " pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.122530 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.122504 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:31.122530 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.122532 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:31.122726 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.122543 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:31.122726 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.122604 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:31.622587589 +0000 UTC m=+3.151932615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:31.123372 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.123339 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m5f\" (UniqueName: \"kubernetes.io/projected/75355291-7dac-483c-ac38-bf786fe91684-kube-api-access-j8m5f\") pod \"ovnkube-node-gzcfs\" (UID: \"75355291-7dac-483c-ac38-bf786fe91684\") " pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.125267 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.125241 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lhf\" (UniqueName: \"kubernetes.io/projected/3caf62a2-ab88-4e90-989d-46df317f1e1c-kube-api-access-x7lhf\") pod \"aws-ebs-csi-driver-node-4xp2k\" (UID: \"3caf62a2-ab88-4e90-989d-46df317f1e1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.125642 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.125615 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxxh\" (UniqueName: \"kubernetes.io/projected/fce84d20-bb40-4e48-ae6a-6c5ce3a255db-kube-api-access-hdxxh\") pod \"multus-additional-cni-plugins-4drnz\" (UID: \"fce84d20-bb40-4e48-ae6a-6c5ce3a255db\") " pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.184962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.184921 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jm68t" Jul 06 19:11:31.198357 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.198323 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd2938_91c2_44d2_b438_25f4c431cccd.slice/crio-38d0bc1ba78ae199c840320eb5114f4989aa6cfb5e652f421aea4049d2888f86 WatchSource:0}: Error finding container 38d0bc1ba78ae199c840320eb5114f4989aa6cfb5e652f421aea4049d2888f86: Status 404 returned error can't find the container with id 38d0bc1ba78ae199c840320eb5114f4989aa6cfb5e652f421aea4049d2888f86 Jul 06 19:11:31.199548 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.199525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:31.209303 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.209276 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4295aa48_adaf_4a8f_b821_38ff2f059993.slice/crio-30315d1344993b2e8410a93ef6fbff2a8b23ec6299d34feab84c7dd9e75473c1 WatchSource:0}: Error finding container 30315d1344993b2e8410a93ef6fbff2a8b23ec6299d34feab84c7dd9e75473c1: Status 404 returned error can't find the container with id 30315d1344993b2e8410a93ef6fbff2a8b23ec6299d34feab84c7dd9e75473c1 Jul 06 19:11:31.210955 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.210933 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5qbrh" Jul 06 19:11:31.216952 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.216931 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-85fsk" Jul 06 19:11:31.224345 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.224317 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7jbm" Jul 06 19:11:31.228657 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.228627 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991de766_d219_4006_a9d9_99cb2b6ac4a3.slice/crio-dfda1c9c1f9f1dd0e7968bcb84438a7e8398872cad2c48164b68f9e50b30080d WatchSource:0}: Error finding container dfda1c9c1f9f1dd0e7968bcb84438a7e8398872cad2c48164b68f9e50b30080d: Status 404 returned error can't find the container with id dfda1c9c1f9f1dd0e7968bcb84438a7e8398872cad2c48164b68f9e50b30080d Jul 06 19:11:31.231714 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.231322 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4drnz" Jul 06 19:11:31.235610 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.235315 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b192ffc_504c_4231_9c5a_efa16fdac630.slice/crio-1f0ae2eb8e50de06f23d2088a9a20655e521f2aff68c1af589aca55ea7c150bf WatchSource:0}: Error finding container 1f0ae2eb8e50de06f23d2088a9a20655e521f2aff68c1af589aca55ea7c150bf: Status 404 returned error can't find the container with id 1f0ae2eb8e50de06f23d2088a9a20655e521f2aff68c1af589aca55ea7c150bf Jul 06 19:11:31.240432 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.240336 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:31.242949 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.242919 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce84d20_bb40_4e48_ae6a_6c5ce3a255db.slice/crio-3cfd1dff32296a339e0de859193753967415ab6c99b10703bae15b4ec8881f30 WatchSource:0}: Error finding container 3cfd1dff32296a339e0de859193753967415ab6c99b10703bae15b4ec8881f30: Status 404 returned error can't find the container with id 3cfd1dff32296a339e0de859193753967415ab6c99b10703bae15b4ec8881f30 Jul 06 19:11:31.249837 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.249814 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" Jul 06 19:11:31.254812 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.254785 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75355291_7dac_483c_ac38_bf786fe91684.slice/crio-d0b47bc24756d76994009b34f26588f8ed3717ed6494eabd65913f34eb91277e WatchSource:0}: Error finding container d0b47bc24756d76994009b34f26588f8ed3717ed6494eabd65913f34eb91277e: Status 404 returned error can't find the container with id d0b47bc24756d76994009b34f26588f8ed3717ed6494eabd65913f34eb91277e Jul 06 19:11:31.256864 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.256844 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t9rlb" Jul 06 19:11:31.272507 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:11:31.272475 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe832039_1f5e_43d8_a376_d94d8be6ae45.slice/crio-3b04ffe359712480015eef9cb575adc370bcf041726e612aa02399f39034c5a7 WatchSource:0}: Error finding container 3b04ffe359712480015eef9cb575adc370bcf041726e612aa02399f39034c5a7: Status 404 returned error can't find the container with id 3b04ffe359712480015eef9cb575adc370bcf041726e612aa02399f39034c5a7 Jul 06 19:11:31.507570 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.507473 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:31.507756 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.507711 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:31.507877 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.507848 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:32.507824468 +0000 UTC m=+4.037169490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:31.608613 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.608569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:31.608799 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.608701 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:31.608799 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.608773 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:32.608753146 +0000 UTC m=+4.138098170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:31.709731 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.709693 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:31.709916 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.709871 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:31.709916 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.709895 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:31.709916 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.709911 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:31.710073 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:31.709997 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:32.709976728 +0000 UTC m=+4.239321760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:31.943569 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.943450 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:31 +0000 UTC" deadline="2027-12-26 18:59:43.435047741 +0000 UTC" Jul 06 19:11:31.943569 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:31.943512 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12911h48m11.491539562s" Jul 06 19:11:32.011476 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.011442 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:32.011654 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.011588 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:32.022410 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.022353 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"d0b47bc24756d76994009b34f26588f8ed3717ed6494eabd65913f34eb91277e"} Jul 06 19:11:32.023543 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.023518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerStarted","Data":"3cfd1dff32296a339e0de859193753967415ab6c99b10703bae15b4ec8881f30"} Jul 06 19:11:32.024766 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.024741 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7jbm" event={"ID":"3b192ffc-504c-4231-9c5a-efa16fdac630","Type":"ContainerStarted","Data":"1f0ae2eb8e50de06f23d2088a9a20655e521f2aff68c1af589aca55ea7c150bf"} Jul 06 19:11:32.028485 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.028395 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85fsk" event={"ID":"991de766-d219-4006-a9d9-99cb2b6ac4a3","Type":"ContainerStarted","Data":"dfda1c9c1f9f1dd0e7968bcb84438a7e8398872cad2c48164b68f9e50b30080d"} Jul 06 19:11:32.030018 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.029997 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qbrh" event={"ID":"3e1ddb84-c2f2-42ce-9438-e2b4bf546381","Type":"ContainerStarted","Data":"b7901abe887862201fb7d72d0292b62cd812432821cefc6d377fe2a739c85a51"} Jul 06 19:11:32.031135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.031096 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghvvr" event={"ID":"4295aa48-adaf-4a8f-b821-38ff2f059993","Type":"ContainerStarted","Data":"30315d1344993b2e8410a93ef6fbff2a8b23ec6299d34feab84c7dd9e75473c1"} Jul 06 19:11:32.032228 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.032209 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" event={"ID":"3caf62a2-ab88-4e90-989d-46df317f1e1c","Type":"ContainerStarted","Data":"a2107f2a92af323814efc54324ea1f1839f0520c94774b10db019a3b33e78b85"} Jul 06 19:11:32.033352 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.033325 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jm68t" event={"ID":"35bd2938-91c2-44d2-b438-25f4c431cccd","Type":"ContainerStarted","Data":"38d0bc1ba78ae199c840320eb5114f4989aa6cfb5e652f421aea4049d2888f86"} Jul 06 19:11:32.035333 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.035281 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t9rlb" event={"ID":"be832039-1f5e-43d8-a376-d94d8be6ae45","Type":"ContainerStarted","Data":"3b04ffe359712480015eef9cb575adc370bcf041726e612aa02399f39034c5a7"} Jul 06 19:11:32.289791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.289690 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:32.441739 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.441708 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:11:32.515746 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.515711 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:32.515950 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.515925 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:32.516027 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.516018 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:34.515995779 +0000 UTC m=+6.045340785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:32.616992 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.616531 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:32.617161 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.616991 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:32.617161 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.617063 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:34.61704355 +0000 UTC m=+6.146388571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:32.718043 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.717998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:32.718258 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.718231 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:32.718321 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.718259 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:32.718321 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.718273 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:32.718461 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:32.718335 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:34.718315457 +0000 UTC m=+6.247660486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:32.944355 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.944209 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:06:31 +0000 UTC" deadline="2028-03-09 05:04:19.55334431 +0000 UTC" Jul 06 19:11:32.944355 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:32.944270 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14673h52m46.60909292s" Jul 06 19:11:33.014452 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:33.012323 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:33.014452 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:33.012476 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:33.014452 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:33.012620 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:33.014452 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:33.012797 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:34.011943 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.011494 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:34.011943 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.011665 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:34.058783 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.058741 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" event={"ID":"3312f4e0b6092b33b85bb20010305980","Type":"ContainerStarted","Data":"d1618d2e36b373b1f6115fe5739506dd3709cb27b098bfd9bdff36ad2e13ccb0"} Jul 06 19:11:34.071607 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.071513 2558 generic.go:356] "Generic (PLEG): container finished" podID="61e1073d4c048b344be043b81c15583d" containerID="b82d0a4129efa0b1ae41336fd491af15374429b773e3df29fb3c98cc6764494f" exitCode=0 Jul 06 19:11:34.071607 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.071578 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" event={"ID":"61e1073d4c048b344be043b81c15583d","Type":"ContainerDied","Data":"b82d0a4129efa0b1ae41336fd491af15374429b773e3df29fb3c98cc6764494f"} Jul 06 19:11:34.101362 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.101300 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-63.ec2.internal" podStartSLOduration=4.101276718 podStartE2EDuration="4.101276718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:34.082003646 +0000 UTC m=+5.611348675" watchObservedRunningTime="2026-07-06 19:11:34.101276718 +0000 UTC m=+5.630621750" Jul 06 19:11:34.539668 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.539623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:34.539855 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.539811 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:34.539923 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.539879 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:38.539859921 +0000 UTC m=+10.069204932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:34.641096 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.640341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:34.641096 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.640541 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:34.641096 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.640611 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:38.640590726 +0000 UTC m=+10.169935731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:34.741841 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:34.741794 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:34.742007 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.741969 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:34.742007 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.741992 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:34.742007 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.742006 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:34.742182 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:34.742071 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:38.742052256 +0000 UTC m=+10.271397265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:35.011804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:35.011702 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:35.011988 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:35.011845 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:35.012513 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:35.012264 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:35.012513 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:35.012356 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:35.082560 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:35.082513 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" event={"ID":"61e1073d4c048b344be043b81c15583d","Type":"ContainerStarted","Data":"74a118da1ca9bca8e05785a441f3075ecf86618e96b601cd6c3535ca3ba4caec"} Jul 06 19:11:36.012442 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:36.012388 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:36.012960 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:36.012563 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:37.012373 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:37.012322 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:37.012550 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:37.012377 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:37.012550 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:37.012485 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:37.012851 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:37.012631 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:38.011932 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:38.011892 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:38.012126 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.012051 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:38.576705 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:38.576663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:38.577158 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.576884 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:38.577158 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.576978 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.576955209 +0000 UTC m=+18.106300236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:38.677307 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:38.677262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:38.677483 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.677411 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:38.677549 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.677485 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.677467541 +0000 UTC m=+18.206812546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:38.778116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:38.778074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:38.778263 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.778235 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:38.778263 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.778251 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:38.778263 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.778262 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:38.778378 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:38.778310 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:46.778297444 +0000 UTC m=+18.307642464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:39.013662 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:39.013580 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:39.014902 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:39.014866 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:39.015232 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:39.015199 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:39.015379 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:39.015352 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:40.012399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:40.012343 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:40.012842 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:40.012522 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:41.013681 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:41.012178 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:41.013681 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:41.012186 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:41.013681 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:41.012322 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:41.013681 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:41.012402 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:42.012684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:42.012159 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:42.012684 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:42.012310 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:43.011598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:43.011562 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:43.011961 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:43.011700 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:43.012162 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:43.012048 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:43.012162 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:43.012126 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:44.012375 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.012170 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:44.012860 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:44.012499 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:44.106654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.106616 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="7aaed1b566ecdfe291a7109e4cccdbd5bce00e0c699c90a54688b8f9444b6c06" exitCode=0 Jul 06 19:11:44.106827 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.106709 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"7aaed1b566ecdfe291a7109e4cccdbd5bce00e0c699c90a54688b8f9444b6c06"} Jul 06 19:11:44.108703 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.108651 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7jbm" event={"ID":"3b192ffc-504c-4231-9c5a-efa16fdac630","Type":"ContainerStarted","Data":"5e821a941b2ffd73840ed27577a06f16cff124a093d0adeffbaceeb8282073fc"} Jul 06 19:11:44.110266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.110227 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-85fsk" event={"ID":"991de766-d219-4006-a9d9-99cb2b6ac4a3","Type":"ContainerStarted","Data":"4dc15beb23b89155a2c324eda102f2b54dca8f02a4ff827ad57791250c5b4943"} Jul 06 19:11:44.112225 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.112198 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5qbrh" event={"ID":"3e1ddb84-c2f2-42ce-9438-e2b4bf546381","Type":"ContainerStarted","Data":"a86ee22d89fc0d33ce06bf8252a073bc79065b07423697b735d0a72cf21c3391"} Jul 06 19:11:44.113711 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.113687 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghvvr" event={"ID":"4295aa48-adaf-4a8f-b821-38ff2f059993","Type":"ContainerStarted","Data":"62aa2dc9245c019ef349dd9657cc4208ccf71cfaedf313f943728a12dcbe9b96"} Jul 06 19:11:44.115175 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.115155 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" event={"ID":"3caf62a2-ab88-4e90-989d-46df317f1e1c","Type":"ContainerStarted","Data":"0b3619a174838428089e8818a400d8e3a99ce68ceca8bcc697f2af0e7433efee"} Jul 06 19:11:44.138325 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.138269 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-63.ec2.internal" podStartSLOduration=14.138250833 podStartE2EDuration="14.138250833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:35.10375262 +0000 UTC m=+6.633097653" watchObservedRunningTime="2026-07-06 19:11:44.138250833 +0000 UTC m=+15.667595862" Jul 06 19:11:44.184255 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.184152 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5qbrh" podStartSLOduration=3.373075737 podStartE2EDuration="15.184136503s" totalImagesPullingTime="11.811060766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:44.156836442 +0000 UTC m=+15.686181469" watchObservedRunningTime="2026-07-06 19:11:44.184136503 +0000 UTC m=+15.713481531" Jul 06 19:11:44.184554 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.184521 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-85fsk" podStartSLOduration=3.363988395 podStartE2EDuration="15.184512133s" totalImagesPullingTime="11.820523738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:44.183899436 +0000 UTC m=+15.713244463" watchObservedRunningTime="2026-07-06 19:11:44.184512133 +0000 UTC m=+15.713857157" Jul 06 19:11:44.212466 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.212404 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r7jbm" podStartSLOduration=3.416359509 podStartE2EDuration="15.212386616s" totalImagesPullingTime="11.796027107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:44.211978625 +0000 UTC m=+15.741323650" watchObservedRunningTime="2026-07-06 19:11:44.212386616 +0000 UTC m=+15.741731641" Jul 06 19:11:44.247698 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.247638 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ghvvr" podStartSLOduration=3.42858477 podStartE2EDuration="15.247621145s" totalImagesPullingTime="11.819036375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:44.24738513 +0000 UTC m=+15.776730154" watchObservedRunningTime="2026-07-06 19:11:44.247621145 +0000 UTC m=+15.776966170" Jul 06 19:11:44.712103 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.712059 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:44.712731 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:44.712712 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:45.011705 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:45.011622 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:45.011866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:45.011622 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:45.011866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:45.011763 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:45.011866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:45.011853 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:45.118656 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:45.118619 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t9rlb" event={"ID":"be832039-1f5e-43d8-a376-d94d8be6ae45","Type":"ContainerStarted","Data":"aceac871ddeab24b68f1bc4e98fe0e1b4c0d89a9f38f11efd2cbffe248a42516"} Jul 06 19:11:45.138877 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:45.138820 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t9rlb" podStartSLOduration=3.371643656 podStartE2EDuration="15.138804376s" totalImagesPullingTime="11.76716072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:45.137367751 +0000 UTC m=+16.666712770" watchObservedRunningTime="2026-07-06 19:11:45.138804376 +0000 UTC m=+16.668149402" Jul 06 19:11:46.012304 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:46.012269 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:46.012485 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.012423 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:46.120304 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:46.120273 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:11:46.650301 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:46.650267 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:46.650488 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.650410 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:46.650488 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.650472 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:02.650456468 +0000 UTC m=+34.179801474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:11:46.750672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:46.750628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:46.750847 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.750809 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:46.750920 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.750864 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:02.750849002 +0000 UTC m=+34.280194005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:11:46.851716 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:46.851680 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:46.851885 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.851845 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:11:46.851885 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.851867 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:11:46.851885 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.851879 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:46.852029 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:46.851935 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:02.851919044 +0000 UTC m=+34.381264050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:11:47.013739 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:47.013664 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:47.013909 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:47.013785 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:47.014162 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:47.014144 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:47.014247 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:47.014227 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:48.011540 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:48.011495 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:48.012119 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:48.011640 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:49.012818 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:49.012585 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:49.013221 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:49.012899 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:49.013221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:49.012686 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:49.013343 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:49.013299 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:50.011565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:50.011528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:50.011727 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:50.011653 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:51.012397 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:51.012360 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:51.012397 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:51.012397 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:51.012886 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:51.012474 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:51.012886 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:51.012591 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:51.696941 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:51.696894 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:51.697290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:51.697033 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:11:51.697595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:51.697566 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghvvr" Jul 06 19:11:52.012288 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:52.012202 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:52.012473 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:52.012338 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:52.770175 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:52.770154 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:11:52.972961 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:52.972866 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:11:52.770170319Z","UUID":"fde24b66-45ef-4c37-80eb-49070849ce76","Handler":null,"Name":"","Endpoint":""} Jul 06 19:11:52.974315 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:52.974262 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:11:52.974315 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:52.974308 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:11:53.012083 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.012060 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:53.012221 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:53.012193 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:53.012417 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.012242 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:53.012417 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:53.012321 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:53.140643 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.140409 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" event={"ID":"3caf62a2-ab88-4e90-989d-46df317f1e1c","Type":"ContainerStarted","Data":"846591c93754d87622dde7a0fefc90056e157dbf95db729505259b1c249f91a1"} Jul 06 19:11:53.142266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.142228 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jm68t" event={"ID":"35bd2938-91c2-44d2-b438-25f4c431cccd","Type":"ContainerStarted","Data":"1101df1d40cf572072fef8ab34ba3bb4e552d22b6b310203f17215acbedbf0de"} Jul 06 19:11:53.145634 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.145615 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:11:53.146008 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.145984 2558 generic.go:356] "Generic (PLEG): container finished" podID="75355291-7dac-483c-ac38-bf786fe91684" containerID="0db627a92379f8707d025d7b35dd1a0493f4b5889706bc09556d8f7eba6b0fa6" exitCode=1 Jul 06 19:11:53.146095 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.146018 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"6369b6a8ada5d675414925d6bf2b5e87064a2349f60d9bb4ad8fe86e0a718cc0"} Jul 06 19:11:53.146136 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.146110 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"ffaad7482e927c992265e4163ca34fe8f1271b4e61b244b0024ce7662f7b0189"} Jul 06 19:11:53.146189 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.146160 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"a3559075e8833ff3d5143bbea7dc9cdc524c5f81e31eb4ce3f110f16a54ca1bb"} Jul 06 19:11:53.146238 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.146211 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerDied","Data":"0db627a92379f8707d025d7b35dd1a0493f4b5889706bc09556d8f7eba6b0fa6"} Jul 06 19:11:53.146282 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.146259 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"7aa12c086fce1061db8a89d217d143e4e3f9ffefd5d34c4d5f76fe48d37e65ec"} Jul 06 19:11:53.166299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:53.166245 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jm68t" podStartSLOduration=2.736463009 podStartE2EDuration="24.166223618s" totalImagesPullingTime="21.429760609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:53.165132444 +0000 UTC m=+24.694477470" watchObservedRunningTime="2026-07-06 19:11:53.166223618 +0000 UTC m=+24.695568644" Jul 06 19:11:54.012326 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:54.012298 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:54.012547 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:54.012453 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:54.150804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:54.150776 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:11:54.151335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:54.151191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"7d99f7642017a5466f7823dfdc81834671e01830765ee0d7be8cc1edcb234d2b"} Jul 06 19:11:55.012219 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:55.012188 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:55.012398 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:55.012192 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:55.012398 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:55.012292 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:55.012398 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:55.012376 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:56.011916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.011880 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:56.012343 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:56.011994 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:56.157841 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.157813 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:11:56.158225 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.158203 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"bad636bb2feb660ff0fc96b4bbd4a6857cc144871d5a0a406ef87859c4ecafea"} Jul 06 19:11:56.159827 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.159802 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="d0124d67acfbd158fa5eb2a03bc824f7bc77569c4262d5072550cb0873baa40d" exitCode=0 Jul 06 19:11:56.159938 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.159883 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"d0124d67acfbd158fa5eb2a03bc824f7bc77569c4262d5072550cb0873baa40d"} Jul 06 19:11:56.161794 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.161776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" event={"ID":"3caf62a2-ab88-4e90-989d-46df317f1e1c","Type":"ContainerStarted","Data":"0bf560cd27a3b245b2c5136931ee12b919af0a67907d82a2380e0318e5b6c540"} Jul 06 19:11:56.217956 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:56.217850 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xp2k" podStartSLOduration=2.15686292 podStartE2EDuration="26.217835726s" totalImagesPullingTime="24.060972806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:56.217691623 +0000 UTC m=+27.747036669" watchObservedRunningTime="2026-07-06 19:11:56.217835726 +0000 UTC m=+27.747180751" Jul 06 19:11:57.011966 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:57.011936 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:57.012384 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:57.012077 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:57.012384 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:57.012149 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:57.012384 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:57.012248 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:58.012522 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.012230 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:58.012522 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:58.012512 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:11:58.170924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.170855 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:11:58.171291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.171267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"8b0e8f71e04f84c899e17ed3e730621604b65de99db6567c781837ec7987983c"} Jul 06 19:11:58.171709 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.171627 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:58.171808 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.171724 2558 scope.go:117] "RemoveContainer" containerID="0db627a92379f8707d025d7b35dd1a0493f4b5889706bc09556d8f7eba6b0fa6" Jul 06 19:11:58.173437 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.173411 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="cc912d4e8ab8a7c4d5ac404a1e292e5a21134e7d03d8025c7c10bab7a4312f1c" exitCode=0 Jul 06 19:11:58.173564 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.173461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"cc912d4e8ab8a7c4d5ac404a1e292e5a21134e7d03d8025c7c10bab7a4312f1c"} Jul 06 19:11:58.188428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:58.188411 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:59.012811 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.012783 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:59.013201 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:59.012858 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:59.013201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.012954 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:59.013201 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:59.013034 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:59.180574 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.180548 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:11:59.180922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.180892 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" event={"ID":"75355291-7dac-483c-ac38-bf786fe91684","Type":"ContainerStarted","Data":"3b836a9869a8875e2b304fbca7de7d211c42fa38bcfe6769dd901b1ae16c4f72"} Jul 06 19:11:59.181252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.181235 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:59.181399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.181383 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:59.207709 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.207682 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:11:59.228176 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.228121 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" podStartSLOduration=7.80235095 podStartE2EDuration="29.228088457s" totalImagesPullingTime="21.425737507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:59.226705206 +0000 UTC m=+30.756050231" watchObservedRunningTime="2026-07-06 19:11:59.228088457 +0000 UTC m=+30.757433485" Jul 06 19:11:59.947636 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.947546 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-22lf6"] Jul 06 19:11:59.947782 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.947681 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:11:59.947846 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:59.947797 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:11:59.948278 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.948257 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8h8p"] Jul 06 19:11:59.948361 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.948341 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:11:59.948434 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:59.948419 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:11:59.950253 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.950231 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7p7g"] Jul 06 19:11:59.950360 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:11:59.950337 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:11:59.950473 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:11:59.950456 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:12:00.185450 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:00.185406 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="b7fa71383586dcbe10b2aca931d5952f81f6507c717fe2038dfbc53eceeab05d" exitCode=0 Jul 06 19:12:00.185832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:00.185476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"b7fa71383586dcbe10b2aca931d5952f81f6507c717fe2038dfbc53eceeab05d"} Jul 06 19:12:02.011791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.011540 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:02.012223 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.011898 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:12:02.012280 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.012258 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:12:02.012368 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.012330 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:12:02.012538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.012465 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:02.012596 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.012540 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:12:02.672987 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.672953 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:12:02.673197 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.673104 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:02.673197 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.673172 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:34.673155944 +0000 UTC m=+66.202500947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:12:02.773373 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.773326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:02.773554 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.773477 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:02.773554 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.773538 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret podName:61c0fbb0-8a79-474c-a770-2c14e44043b0 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:34.773520996 +0000 UTC m=+66.302866001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret") pod "global-pull-secret-syncer-x8h8p" (UID: "61c0fbb0-8a79-474c-a770-2c14e44043b0") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:12:02.874004 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:02.873917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:02.874169 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.874107 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:12:02.874169 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.874130 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:12:02.874169 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.874141 2558 projected.go:196] Error preparing data for projected volume kube-api-access-h4zqp for pod openshift-network-diagnostics/network-check-target-22lf6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:02.874323 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:02.874208 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp podName:4d26d041-686f-4ba7-ba42-b4a28a217829 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:34.874190703 +0000 UTC m=+66.403535720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4zqp" (UniqueName: "kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp") pod "network-check-target-22lf6" (UID: "4d26d041-686f-4ba7-ba42-b4a28a217829") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:12:04.011585 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:04.011544 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:04.012094 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:04.011570 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:12:04.012094 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:04.011665 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:04.012094 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:04.011672 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22lf6" podUID="4d26d041-686f-4ba7-ba42-b4a28a217829" Jul 06 19:12:04.012094 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:04.011743 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:12:04.012094 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:04.011840 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8h8p" podUID="61c0fbb0-8a79-474c-a770-2c14e44043b0" Jul 06 19:12:05.253589 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.253562 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-63.ec2.internal" event="NodeReady" Jul 06 19:12:05.254157 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.253716 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:12:05.304414 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.304379 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:12:05.331747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.331710 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4rglq"] Jul 06 19:12:05.331923 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.331837 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.334241 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.334221 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:12:05.336508 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.336483 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lwjk9\"" Jul 06 19:12:05.339380 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.339344 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:12:05.340280 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.340028 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:12:05.344045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.343992 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.346923 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.346902 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:12:05.347039 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.347025 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g22xl\"" Jul 06 19:12:05.347257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.347242 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:12:05.363677 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.363652 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:12:05.364441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.364417 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4rglq"] Jul 06 19:12:05.379738 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.379710 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tckbk"] Jul 06 19:12:05.395413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395228 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395295 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be2f178-730b-4a7e-8a12-a163c0649821-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.395413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395367 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395394 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395597 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395631 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdmz\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.395844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.395655 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.400063 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.400035 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5zn82"] Jul 06 19:12:05.400183 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.400158 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.403593 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.403306 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:12:05.403593 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.403329 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:12:05.403593 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.403558 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:12:05.419845 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.419819 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.426364 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.426279 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tckbk"] Jul 06 19:12:05.426474 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.426432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:12:05.426674 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.426656 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:12:05.427105 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.426946 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zsvz6\"" Jul 06 19:12:05.428074 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.428052 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zn82"] Jul 06 19:12:05.429094 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.429076 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:12:05.496796 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.496753 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be2f178-730b-4a7e-8a12-a163c0649821-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.496796 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.496799 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3071cbb5-ad05-492b-a327-1cd14a91e3c4-cert\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.497042 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.496826 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497042 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.496960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497042 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497081 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497133 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497162 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdmz\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497217 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-config-volume\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497274 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497328 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9m6\" (UniqueName: \"kubernetes.io/projected/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-kube-api-access-nc9m6\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-tmp-dir\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497409 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8qp\" (UniqueName: \"kubernetes.io/projected/3071cbb5-ad05-492b-a327-1cd14a91e3c4-kube-api-access-qx8qp\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.497471 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497474 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.497612 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.497566 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:05.997543797 +0000 UTC m=+37.526888799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:05.498019 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.497673 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:05.498019 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497679 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.498019 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.497687 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:05.498019 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.497803 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be2f178-730b-4a7e-8a12-a163c0649821-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:05.498019 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.497878 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:05.997853247 +0000 UTC m=+37.527198251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:05.498704 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.498681 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.502287 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.502264 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:12:05.502502 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.502320 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.502690 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.502665 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.519387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.519333 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.529990 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.529959 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdmz\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:05.598135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-config-volume\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.598135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598111 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.598386 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.598227 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:05.598386 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:05.598283 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:06.098268061 +0000 UTC m=+37.627613069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:05.598386 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598224 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9m6\" (UniqueName: \"kubernetes.io/projected/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-kube-api-access-nc9m6\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.598554 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-tmp-dir\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.598554 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8qp\" (UniqueName: \"kubernetes.io/projected/3071cbb5-ad05-492b-a327-1cd14a91e3c4-kube-api-access-qx8qp\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.598554 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598459 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3071cbb5-ad05-492b-a327-1cd14a91e3c4-cert\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.598761 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598739 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-config-volume\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.598761 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.598750 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-tmp-dir\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.601128 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.601107 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3071cbb5-ad05-492b-a327-1cd14a91e3c4-cert\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.613957 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.613933 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9m6\" (UniqueName: \"kubernetes.io/projected/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-kube-api-access-nc9m6\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:05.615061 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.615040 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8qp\" (UniqueName: \"kubernetes.io/projected/3071cbb5-ad05-492b-a327-1cd14a91e3c4-kube-api-access-qx8qp\") pod \"ingress-canary-5zn82\" (UID: \"3071cbb5-ad05-492b-a327-1cd14a91e3c4\") " pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.729671 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.729632 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zn82" Jul 06 19:12:05.809895 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.809847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghvvr_4295aa48-adaf-4a8f-b821-38ff2f059993/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:05.928391 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:05.928283 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghvvr_4295aa48-adaf-4a8f-b821-38ff2f059993/konnectivity-agent/0.log" limit=1024 Jul 06 19:12:06.002038 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.001998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:06.002241 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.002097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:06.002241 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.002175 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:06.002241 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.002199 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:06.002241 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.002214 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:06.002431 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.002276 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:07.00225219 +0000 UTC m=+38.531597216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:06.002431 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.002297 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:07.002287152 +0000 UTC m=+38.531632156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:06.011565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.011534 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:06.011565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.011563 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:12:06.011794 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.011545 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:06.014782 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.014757 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:06.015052 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.015035 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:06.015313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.015287 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:06.015756 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.015734 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:12:06.015852 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.015798 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:06.016444 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.016426 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:06.102765 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.102726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:06.102944 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.102855 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:06.102944 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:06.102925 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:07.102904696 +0000 UTC m=+38.632249700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:06.492242 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:06.492211 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zn82"] Jul 06 19:12:06.495285 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:12:06.495261 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3071cbb5_ad05_492b_a327_1cd14a91e3c4.slice/crio-2aed7150ba2ab670503e3507e00c39e47d6de4958ff0eca61b1aca9fc618c0d0 WatchSource:0}: Error finding container 2aed7150ba2ab670503e3507e00c39e47d6de4958ff0eca61b1aca9fc618c0d0: Status 404 returned error can't find the container with id 2aed7150ba2ab670503e3507e00c39e47d6de4958ff0eca61b1aca9fc618c0d0 Jul 06 19:12:07.010474 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.010386 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:07.010626 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.010481 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:07.010626 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.010550 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:07.010626 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.010579 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:07.010626 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.010590 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:07.010626 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.010622 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:09.010603589 +0000 UTC m=+40.539948606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:07.010801 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.010638 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:09.010632604 +0000 UTC m=+40.539977606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:07.111260 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.111232 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:07.111453 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.111401 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:07.111502 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:07.111462 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:09.11144453 +0000 UTC m=+40.640789553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:07.207720 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.207690 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="a52523c4731be5383225dfd9396c02ed2ee65521b89d6b427a3bb6b6e0f2ba27" exitCode=0 Jul 06 19:12:07.207878 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.207775 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"a52523c4731be5383225dfd9396c02ed2ee65521b89d6b427a3bb6b6e0f2ba27"} Jul 06 19:12:07.208938 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:07.208919 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zn82" event={"ID":"3071cbb5-ad05-492b-a327-1cd14a91e3c4","Type":"ContainerStarted","Data":"2aed7150ba2ab670503e3507e00c39e47d6de4958ff0eca61b1aca9fc618c0d0"} Jul 06 19:12:08.214250 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:08.214218 2558 generic.go:356] "Generic (PLEG): container finished" podID="fce84d20-bb40-4e48-ae6a-6c5ce3a255db" containerID="7ac641ee5ed8ad8110207af3c5467ebb7ef23249e9f7537068a76681bebc0481" exitCode=0 Jul 06 19:12:08.214629 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:08.214270 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerDied","Data":"7ac641ee5ed8ad8110207af3c5467ebb7ef23249e9f7537068a76681bebc0481"} Jul 06 19:12:09.027024 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:09.026986 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:09.027203 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:09.027067 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:09.027203 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.027129 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:09.027203 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.027146 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:09.027203 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.027164 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:09.027203 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.027196 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:13.027180861 +0000 UTC m=+44.556525864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:09.027405 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.027217 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:13.027202049 +0000 UTC m=+44.556547052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:09.128647 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:09.128393 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:09.128806 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.128543 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:09.128806 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:09.128780 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:13.128763926 +0000 UTC m=+44.658108945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:09.221113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:09.221075 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4drnz" event={"ID":"fce84d20-bb40-4e48-ae6a-6c5ce3a255db","Type":"ContainerStarted","Data":"831aedb13f80274f289f03d3f5d53b30498d8ddfa3c7021afb1ecbea2c2c1192"} Jul 06 19:12:09.253071 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:09.253014 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4drnz" podStartSLOduration=6.990909071 podStartE2EDuration="39.25299675s" totalImagesPullingTime="32.262087679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:09.252050168 +0000 UTC m=+40.781395187" watchObservedRunningTime="2026-07-06 19:12:09.25299675 +0000 UTC m=+40.782341772" Jul 06 19:12:10.225264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:10.225222 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zn82" event={"ID":"3071cbb5-ad05-492b-a327-1cd14a91e3c4","Type":"ContainerStarted","Data":"894e948c84f68a592903df699d5bec59b5ab3fd97ec6b88cb1b507deeadb99f2"} Jul 06 19:12:10.248988 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:10.248942 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5zn82" podStartSLOduration=1.884852488 podStartE2EDuration="5.248926901s" totalImagesPullingTime="3.364074413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:10.248669039 +0000 UTC m=+41.778014066" watchObservedRunningTime="2026-07-06 19:12:10.248926901 +0000 UTC m=+41.778271972" Jul 06 19:12:13.064009 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:13.063975 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:13.064049 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.064128 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.064203 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:21.064188106 +0000 UTC m=+52.593533109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.064133 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.064241 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:13.064387 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.064292 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:21.064276627 +0000 UTC m=+52.593621649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:13.164561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:13.164528 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:13.164716 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.164652 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:13.164716 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:13.164704 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:21.164690997 +0000 UTC m=+52.694036000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:21.128249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:21.128209 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:21.128310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.128379 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.128447 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.128458 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:37.128441424 +0000 UTC m=+68.657786426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.128464 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:21.128866 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.128504 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:37.128491228 +0000 UTC m=+68.657836235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:21.228984 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:21.228946 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:21.229145 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.229100 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:21.229183 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:21.229161 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:12:37.229146868 +0000 UTC m=+68.758491870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:31.199407 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:31.199374 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gzcfs" Jul 06 19:12:34.735791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.735741 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:12:34.738640 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.738623 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:12:34.746906 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:34.746884 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:12:34.746964 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:34.746944 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:38.746928909 +0000 UTC m=+130.276273912 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : secret "metrics-daemon-secret" not found Jul 06 19:12:34.836328 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.836285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:34.839320 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.839300 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:12:34.850618 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.850586 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61c0fbb0-8a79-474c-a770-2c14e44043b0-original-pull-secret\") pod \"global-pull-secret-syncer-x8h8p\" (UID: \"61c0fbb0-8a79-474c-a770-2c14e44043b0\") " pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:34.937105 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.937063 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:34.941760 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.941741 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:12:34.951127 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.951104 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:12:34.960882 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:34.960859 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zqp\" (UniqueName: \"kubernetes.io/projected/4d26d041-686f-4ba7-ba42-b4a28a217829-kube-api-access-h4zqp\") pod \"network-check-target-22lf6\" (UID: \"4d26d041-686f-4ba7-ba42-b4a28a217829\") " pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:35.123652 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.123560 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8h8p" Jul 06 19:12:35.133284 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.133261 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-422bs\"" Jul 06 19:12:35.140537 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.140508 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:35.271770 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.271740 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8h8p"] Jul 06 19:12:35.274945 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:12:35.274917 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c0fbb0_8a79_474c_a770_2c14e44043b0.slice/crio-36d98a3190c171d991a064060217155c7c2fe72c549c6fe087926bf019fcfbc9 WatchSource:0}: Error finding container 36d98a3190c171d991a064060217155c7c2fe72c549c6fe087926bf019fcfbc9: Status 404 returned error can't find the container with id 36d98a3190c171d991a064060217155c7c2fe72c549c6fe087926bf019fcfbc9 Jul 06 19:12:35.286837 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.286811 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8h8p" event={"ID":"61c0fbb0-8a79-474c-a770-2c14e44043b0","Type":"ContainerStarted","Data":"36d98a3190c171d991a064060217155c7c2fe72c549c6fe087926bf019fcfbc9"} Jul 06 19:12:35.293174 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:35.293153 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-22lf6"] Jul 06 19:12:35.296888 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:12:35.296866 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d26d041_686f_4ba7_ba42_b4a28a217829.slice/crio-aae5ec2f0aeabf6ee0b06e810e0a387fcb41569dd832868ed8e64560da982733 WatchSource:0}: Error finding container aae5ec2f0aeabf6ee0b06e810e0a387fcb41569dd832868ed8e64560da982733: Status 404 returned error can't find the container with id aae5ec2f0aeabf6ee0b06e810e0a387fcb41569dd832868ed8e64560da982733 Jul 06 19:12:36.292922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:36.292883 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-22lf6" event={"ID":"4d26d041-686f-4ba7-ba42-b4a28a217829","Type":"ContainerStarted","Data":"aae5ec2f0aeabf6ee0b06e810e0a387fcb41569dd832868ed8e64560da982733"} Jul 06 19:12:37.154735 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:37.154698 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:12:37.154924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:37.154771 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:12:37.154924 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.154881 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:12:37.154924 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.154911 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:12:37.155079 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.154963 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:09.154947239 +0000 UTC m=+100.684292243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:12:37.155079 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.154886 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:12:37.155079 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.155031 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:09.155018942 +0000 UTC m=+100.684363944 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:12:37.255950 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:37.255919 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:12:37.256112 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.256072 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:12:37.256150 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:12:37.256134 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:13:09.256119766 +0000 UTC m=+100.785464770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:12:41.307047 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:41.307005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8h8p" event={"ID":"61c0fbb0-8a79-474c-a770-2c14e44043b0","Type":"ContainerStarted","Data":"b8269ba48772714c06b3d900e2a1b2bef3ef3144e717dd2054646961dbf842c8"} Jul 06 19:12:41.308436 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:41.308413 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-22lf6" event={"ID":"4d26d041-686f-4ba7-ba42-b4a28a217829","Type":"ContainerStarted","Data":"49b77a10fae4f4dd85563f768cd101a3c1547d8ecde95e1a1b0b9af14663766b"} Jul 06 19:12:41.308573 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:41.308554 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:12:41.333117 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:41.333066 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x8h8p" podStartSLOduration=66.078034956 podStartE2EDuration="1m11.333025438s" totalImagesPullingTime="5.254990482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:41.33256034 +0000 UTC m=+72.861905366" watchObservedRunningTime="2026-07-06 19:12:41.333025438 +0000 UTC m=+72.862370463" Jul 06 19:12:41.356651 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:12:41.356609 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-22lf6" podStartSLOduration=66.127842636 podStartE2EDuration="1m11.356591931s" totalImagesPullingTime="5.228749295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:41.356572493 +0000 UTC m=+72.885917528" watchObservedRunningTime="2026-07-06 19:12:41.356591931 +0000 UTC m=+72.885937008" Jul 06 19:13:09.185695 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:13:09.185577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:13:09.185695 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:13:09.185625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:13:09.186190 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.185729 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:13:09.186190 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.185748 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:13:09.186190 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.185802 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:13:09.186190 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.185821 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:14:13.185800848 +0000 UTC m=+164.715145851 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:13:09.186190 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.185852 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:14:13.185839425 +0000 UTC m=+164.715184428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:13:09.286125 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:13:09.286083 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:13:09.286290 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.286240 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:13:09.286336 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:09.286325 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:14:13.286304219 +0000 UTC m=+164.815649222 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:13:12.313530 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:13:12.313500 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-22lf6" Jul 06 19:13:38.804472 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:13:38.804438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:13:38.804928 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:38.804558 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:13:38.804928 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:13:38.804612 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:15:40.804597471 +0000 UTC m=+252.333942474 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : secret "metrics-daemon-secret" not found Jul 06 19:14:08.344092 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:08.344049 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" Jul 06 19:14:08.354242 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:08.354200 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" podUID="7be2f178-730b-4a7e-8a12-a163c0649821" Jul 06 19:14:08.411445 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:08.411419 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tckbk" podUID="c0d423a6-8e0c-44a4-a3e2-4a8a76e80633" Jul 06 19:14:08.516284 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:08.516254 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:14:08.516284 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:08.516284 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tckbk" Jul 06 19:14:08.516506 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:08.516465 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:14:09.036087 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:09.036052 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:14:13.243097 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:13.243047 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:13.243128 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.243212 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.243225 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.243229 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56bc5db5bb-ttfgp: secret "image-registry-tls" not found Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.243292 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls podName:a328252d-e23e-4e49-9e96-7f753a7d9bd3 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:15.243273437 +0000 UTC m=+286.772618454 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls") pod "image-registry-56bc5db5bb-ttfgp" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3") : secret "image-registry-tls" not found Jul 06 19:14:13.243558 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.243308 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:15.243299346 +0000 UTC m=+286.772644356 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:14:13.343665 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:13.343637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:14:13.343838 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.343761 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:14:13.343838 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:14:13.343809 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls podName:c0d423a6-8e0c-44a4-a3e2-4a8a76e80633 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:15.343795634 +0000 UTC m=+286.873140640 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls") pod "dns-default-tckbk" (UID: "c0d423a6-8e0c-44a4-a3e2-4a8a76e80633") : secret "dns-default-metrics-tls" not found Jul 06 19:14:20.011710 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:14:20.011610 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:15:40.849538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:40.849487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:15:40.850030 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:40.849642 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 19:15:40.850030 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:40.849725 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs podName:c193a56a-2f2c-4eac-86bd-e053d5b7bd57 nodeName:}" failed. No retries permitted until 2026-07-06 19:17:42.849708238 +0000 UTC m=+374.379053245 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs") pod "network-metrics-daemon-q7p7g" (UID: "c193a56a-2f2c-4eac-86bd-e053d5b7bd57") : secret "metrics-daemon-secret" not found Jul 06 19:15:56.540473 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.540391 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c"] Jul 06 19:15:56.543292 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.543271 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.547714 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.547674 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.548598 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.548578 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kfvrc\"" Jul 06 19:15:56.550051 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.550030 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.550152 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.550077 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 19:15:56.550152 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.550097 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 19:15:56.563395 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.563372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c"] Jul 06 19:15:56.639198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.639167 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q"] Jul 06 19:15:56.642091 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.642068 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.642325 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.642303 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh"] Jul 06 19:15:56.644623 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.644596 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6klq7\"" Jul 06 19:15:56.644712 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.644649 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.644762 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.644649 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 19:15:56.645011 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.644988 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.645226 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.645187 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" Jul 06 19:15:56.645898 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.645879 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4"] Jul 06 19:15:56.646981 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.646959 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"network-diagnostics-dockercfg-ch8hp\" is forbidden: User \"system:node:ip-10-0-143-63.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-143-63.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ch8hp\"" type="*v1.Secret" Jul 06 19:15:56.648545 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.648528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.650232 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.650210 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-63.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-143-63.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:15:56.654207 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.654192 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 19:15:56.660289 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.660265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3528c111-3fde-4c3a-921c-744cfc1c4ade-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.660459 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.660341 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpcm\" (UniqueName: \"kubernetes.io/projected/3528c111-3fde-4c3a-921c-744cfc1c4ade-kube-api-access-wrpcm\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.660459 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.660393 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.666552 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.666534 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.666641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.666594 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dmqfc\"" Jul 06 19:15:56.666641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.666543 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 19:15:56.667377 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.667344 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh"] Jul 06 19:15:56.674341 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.674324 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q"] Jul 06 19:15:56.695589 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.695566 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4"] Jul 06 19:15:56.744589 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.744559 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc"] Jul 06 19:15:56.747484 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.747469 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" Jul 06 19:15:56.750874 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.750853 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 19:15:56.750990 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.750927 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:15:56.751054 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.750988 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sw7vn\"" Jul 06 19:15:56.761176 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfv6r\" (UniqueName: \"kubernetes.io/projected/79110540-89d9-4f26-a37d-2fa65bcc4922-kube-api-access-hfv6r\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.761272 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761182 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.761272 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761198 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.761272 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3528c111-3fde-4c3a-921c-744cfc1c4ade-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.761411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761287 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq7l\" (UniqueName: \"kubernetes.io/projected/5ee680c3-a91f-4d2a-b951-e9b9a643e8ad-kube-api-access-cpq7l\") pod \"network-check-source-989f8b8dc-kpmsh\" (UID: \"5ee680c3-a91f-4d2a-b951-e9b9a643e8ad\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" Jul 06 19:15:56.761411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761330 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpcm\" (UniqueName: \"kubernetes.io/projected/3528c111-3fde-4c3a-921c-744cfc1c4ade-kube-api-access-wrpcm\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.761411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761370 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97gf\" (UniqueName: \"kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.761411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761391 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.761411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761409 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.761621 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.761508 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:56.761621 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.761569 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:15:57.261554247 +0000 UTC m=+268.790899250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:56.761893 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.761874 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3528c111-3fde-4c3a-921c-744cfc1c4ade-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.763609 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.763593 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc"] Jul 06 19:15:56.794863 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.794811 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpcm\" (UniqueName: \"kubernetes.io/projected/3528c111-3fde-4c3a-921c-744cfc1c4ade-kube-api-access-wrpcm\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:56.862083 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862055 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxx7\" (UniqueName: \"kubernetes.io/projected/2e8cb9c4-b83f-4ff7-80d9-2485abe1096b-kube-api-access-vhxx7\") pod \"volume-data-source-validator-5d456fdbcd-vtrnc\" (UID: \"2e8cb9c4-b83f-4ff7-80d9-2485abe1096b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" Jul 06 19:15:56.862249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfv6r\" (UniqueName: \"kubernetes.io/projected/79110540-89d9-4f26-a37d-2fa65bcc4922-kube-api-access-hfv6r\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.862249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.862249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.862249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862180 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq7l\" (UniqueName: \"kubernetes.io/projected/5ee680c3-a91f-4d2a-b951-e9b9a643e8ad-kube-api-access-cpq7l\") pod \"network-check-source-989f8b8dc-kpmsh\" (UID: \"5ee680c3-a91f-4d2a-b951-e9b9a643e8ad\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" Jul 06 19:15:56.862249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862227 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97gf\" (UniqueName: \"kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.862518 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862260 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.862518 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.862381 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:15:56.862518 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:56.862441 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls podName:79110540-89d9-4f26-a37d-2fa65bcc4922 nodeName:}" failed. No retries permitted until 2026-07-06 19:15:57.362419076 +0000 UTC m=+268.891764081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29z7q" (UID: "79110540-89d9-4f26-a37d-2fa65bcc4922") : secret "samples-operator-tls" not found Jul 06 19:15:56.862743 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.862725 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.864568 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.864548 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:56.876252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.876227 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq7l\" (UniqueName: \"kubernetes.io/projected/5ee680c3-a91f-4d2a-b951-e9b9a643e8ad-kube-api-access-cpq7l\") pod \"network-check-source-989f8b8dc-kpmsh\" (UID: \"5ee680c3-a91f-4d2a-b951-e9b9a643e8ad\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" Jul 06 19:15:56.876339 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.876301 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfv6r\" (UniqueName: \"kubernetes.io/projected/79110540-89d9-4f26-a37d-2fa65bcc4922-kube-api-access-hfv6r\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:56.962836 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.962784 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxx7\" (UniqueName: \"kubernetes.io/projected/2e8cb9c4-b83f-4ff7-80d9-2485abe1096b-kube-api-access-vhxx7\") pod \"volume-data-source-validator-5d456fdbcd-vtrnc\" (UID: \"2e8cb9c4-b83f-4ff7-80d9-2485abe1096b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" Jul 06 19:15:56.973663 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:56.973632 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxx7\" (UniqueName: \"kubernetes.io/projected/2e8cb9c4-b83f-4ff7-80d9-2485abe1096b-kube-api-access-vhxx7\") pod \"volume-data-source-validator-5d456fdbcd-vtrnc\" (UID: \"2e8cb9c4-b83f-4ff7-80d9-2485abe1096b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" Jul 06 19:15:57.056782 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.056692 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" Jul 06 19:15:57.179726 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.179694 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc"] Jul 06 19:15:57.184923 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:15:57.184894 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8cb9c4_b83f_4ff7_80d9_2485abe1096b.slice/crio-71d43e5ac5f0072fb30613cfb9732567f8673666686a911b4c0831d538f6d442 WatchSource:0}: Error finding container 71d43e5ac5f0072fb30613cfb9732567f8673666686a911b4c0831d538f6d442: Status 404 returned error can't find the container with id 71d43e5ac5f0072fb30613cfb9732567f8673666686a911b4c0831d538f6d442 Jul 06 19:15:57.265507 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.265472 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:57.265669 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.265616 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:57.265717 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.265688 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.265666058 +0000 UTC m=+269.795011061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:57.365949 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.365856 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:57.366081 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.366009 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:15:57.366081 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.366070 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls podName:79110540-89d9-4f26-a37d-2fa65bcc4922 nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.366056375 +0000 UTC m=+269.895401381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29z7q" (UID: "79110540-89d9-4f26-a37d-2fa65bcc4922") : secret "samples-operator-tls" not found Jul 06 19:15:57.766629 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.766598 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" event={"ID":"2e8cb9c4-b83f-4ff7-80d9-2485abe1096b","Type":"ContainerStarted","Data":"71d43e5ac5f0072fb30613cfb9732567f8673666686a911b4c0831d538f6d442"} Jul 06 19:15:57.874801 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.874762 2558 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 06 19:15:57.874801 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.874804 2558 projected.go:196] Error preparing data for projected volume kube-api-access-g97gf for pod openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4: failed to sync configmap cache: timed out waiting for the condition Jul 06 19:15:57.875031 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:57.874869 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf podName:a54efb5e-c0ac-4ab6-99c4-0d833ab26d30 nodeName:}" failed. No retries permitted until 2026-07-06 19:15:58.374853923 +0000 UTC m=+269.904198926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g97gf" (UniqueName: "kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf") pod "kube-storage-version-migrator-operator-846bf668f7-98lq4" (UID: "a54efb5e-c0ac-4ab6-99c4-0d833ab26d30") : failed to sync configmap cache: timed out waiting for the condition Jul 06 19:15:57.904856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.904823 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 19:15:57.959386 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.959336 2558 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" secret="" err="failed to sync secret cache: timed out waiting for the condition" Jul 06 19:15:57.959545 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:57.959407 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" Jul 06 19:15:58.098922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.098887 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh"] Jul 06 19:15:58.102534 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:15:58.102504 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee680c3_a91f_4d2a_b951_e9b9a643e8ad.slice/crio-83f207b85697cba74b724e2c7c7404fa2ac74ccaf942abf38a22fe975ed418ae WatchSource:0}: Error finding container 83f207b85697cba74b724e2c7c7404fa2ac74ccaf942abf38a22fe975ed418ae: Status 404 returned error can't find the container with id 83f207b85697cba74b724e2c7c7404fa2ac74ccaf942abf38a22fe975ed418ae Jul 06 19:15:58.211745 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.211718 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ch8hp\"" Jul 06 19:15:58.273141 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.273107 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:15:58.273290 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:58.273264 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:58.273343 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:58.273333 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:16:00.273317563 +0000 UTC m=+271.802662565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:15:58.374641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.374559 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:15:58.374785 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:58.374709 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:15:58.374785 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:15:58.374773 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls podName:79110540-89d9-4f26-a37d-2fa65bcc4922 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:00.374755197 +0000 UTC m=+271.904100200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29z7q" (UID: "79110540-89d9-4f26-a37d-2fa65bcc4922") : secret "samples-operator-tls" not found Jul 06 19:15:58.475503 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.475472 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97gf\" (UniqueName: \"kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:58.478238 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.478216 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97gf\" (UniqueName: \"kubernetes.io/projected/a54efb5e-c0ac-4ab6-99c4-0d833ab26d30-kube-api-access-g97gf\") pod \"kube-storage-version-migrator-operator-846bf668f7-98lq4\" (UID: \"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:58.763819 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.763799 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" Jul 06 19:15:58.772208 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.772177 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" event={"ID":"2e8cb9c4-b83f-4ff7-80d9-2485abe1096b","Type":"ContainerStarted","Data":"bf3258c472bb0966f93a7c784f40b3c9be1cde3fdc0dcc817072a1219cc52497"} Jul 06 19:15:58.774005 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.773947 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" event={"ID":"5ee680c3-a91f-4d2a-b951-e9b9a643e8ad","Type":"ContainerStarted","Data":"bc3f88894f5ce5e24fc690b1f8be53b06800e173c1e612fc53491fad80551742"} Jul 06 19:15:58.774005 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.773987 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" event={"ID":"5ee680c3-a91f-4d2a-b951-e9b9a643e8ad","Type":"ContainerStarted","Data":"83f207b85697cba74b724e2c7c7404fa2ac74ccaf942abf38a22fe975ed418ae"} Jul 06 19:15:58.790392 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.790324 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-vtrnc" podStartSLOduration=1.29088407 podStartE2EDuration="2.790308227s" totalImagesPullingTime="1.499424157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:58.789225291 +0000 UTC m=+270.318570317" watchObservedRunningTime="2026-07-06 19:15:58.790308227 +0000 UTC m=+270.319653253" Jul 06 19:15:58.809082 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.808257 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-kpmsh" podStartSLOduration=2.808240117 podStartE2EDuration="2.808240117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:58.807635607 +0000 UTC m=+270.336980637" watchObservedRunningTime="2026-07-06 19:15:58.808240117 +0000 UTC m=+270.337585166" Jul 06 19:15:58.899846 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:58.899812 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4"] Jul 06 19:15:58.902804 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:15:58.902773 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54efb5e_c0ac_4ab6_99c4_0d833ab26d30.slice/crio-a486e7cf59b2f381675edfc9ffbca905f7ebe12484fac6dc45dc6533ff3e0fc5 WatchSource:0}: Error finding container a486e7cf59b2f381675edfc9ffbca905f7ebe12484fac6dc45dc6533ff3e0fc5: Status 404 returned error can't find the container with id a486e7cf59b2f381675edfc9ffbca905f7ebe12484fac6dc45dc6533ff3e0fc5 Jul 06 19:15:59.778708 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:15:59.778666 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" event={"ID":"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30","Type":"ContainerStarted","Data":"a486e7cf59b2f381675edfc9ffbca905f7ebe12484fac6dc45dc6533ff3e0fc5"} Jul 06 19:16:00.291144 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:00.291101 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:00.291330 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:00.291268 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:00.291416 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:00.291362 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.291327947 +0000 UTC m=+275.820672950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:00.392510 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:00.392472 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:16:00.392680 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:00.392634 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:16:00.392725 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:00.392700 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls podName:79110540-89d9-4f26-a37d-2fa65bcc4922 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.392682548 +0000 UTC m=+275.922027550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29z7q" (UID: "79110540-89d9-4f26-a37d-2fa65bcc4922") : secret "samples-operator-tls" not found Jul 06 19:16:02.438497 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.438463 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fxmgm"] Jul 06 19:16:02.441700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.441673 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.443841 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.443823 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:16:02.444773 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.444757 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:16:02.444861 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.444783 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:16:02.444861 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.444761 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:16:02.444976 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.444757 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2n6g9\"" Jul 06 19:16:02.454914 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.454889 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fxmgm"] Jul 06 19:16:02.610896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.610864 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.611053 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.610910 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60123861-df62-4e28-8192-5ec05132a8d8-data-volume\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.611053 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.610989 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.611119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.611073 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60123861-df62-4e28-8192-5ec05132a8d8-crio-socket\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.611158 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.611137 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnw9\" (UniqueName: \"kubernetes.io/projected/60123861-df62-4e28-8192-5ec05132a8d8-kube-api-access-qqnw9\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712433 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnw9\" (UniqueName: \"kubernetes.io/projected/60123861-df62-4e28-8192-5ec05132a8d8-kube-api-access-qqnw9\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712433 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712660 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712439 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60123861-df62-4e28-8192-5ec05132a8d8-data-volume\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712660 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:02.712524 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:02.712660 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712595 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712660 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:02.712615 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.21259447 +0000 UTC m=+274.741939489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:02.712867 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:02.712673 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.712867 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712699 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60123861-df62-4e28-8192-5ec05132a8d8-crio-socket\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712867 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:02.712709 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:03.212697598 +0000 UTC m=+274.742042611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:02.712867 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712755 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60123861-df62-4e28-8192-5ec05132a8d8-crio-socket\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.712867 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.712807 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60123861-df62-4e28-8192-5ec05132a8d8-data-volume\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.725041 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.725013 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnw9\" (UniqueName: \"kubernetes.io/projected/60123861-df62-4e28-8192-5ec05132a8d8-kube-api-access-qqnw9\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:02.787992 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.787944 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" event={"ID":"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30","Type":"ContainerStarted","Data":"be9e4c8bbc020e93a08a8c93ff806462530d8d657272cf037255cc77ec6c7d3f"} Jul 06 19:16:02.805709 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:02.805642 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" podStartSLOduration=3.974388686 podStartE2EDuration="6.805625526s" totalImagesPullingTime="2.83123684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:02.804744428 +0000 UTC m=+274.334089476" watchObservedRunningTime="2026-07-06 19:16:02.805625526 +0000 UTC m=+274.334970529" Jul 06 19:16:03.217324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.217293 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:03.217512 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.217392 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:03.217512 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:03.217427 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:03.217512 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:03.217488 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:03.217611 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:03.217490 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.217475345 +0000 UTC m=+275.746820352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:03.217611 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:03.217546 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:04.21753421 +0000 UTC m=+275.746879214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:03.598972 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.598892 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2"] Jul 06 19:16:03.602549 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.602522 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" Jul 06 19:16:03.605501 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.605481 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 19:16:03.605632 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.605520 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6bkr5\"" Jul 06 19:16:03.606020 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.606006 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 19:16:03.614106 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.614086 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2"] Jul 06 19:16:03.721116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.721085 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgmc\" (UniqueName: \"kubernetes.io/projected/8801a073-e65e-40ef-83f6-4dd35c3fdf5c-kube-api-access-tcgmc\") pod \"migrator-d57d58cc9-ldjm2\" (UID: \"8801a073-e65e-40ef-83f6-4dd35c3fdf5c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" Jul 06 19:16:03.822098 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.822059 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgmc\" (UniqueName: \"kubernetes.io/projected/8801a073-e65e-40ef-83f6-4dd35c3fdf5c-kube-api-access-tcgmc\") pod \"migrator-d57d58cc9-ldjm2\" (UID: \"8801a073-e65e-40ef-83f6-4dd35c3fdf5c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" Jul 06 19:16:03.829713 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.829672 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgmc\" (UniqueName: \"kubernetes.io/projected/8801a073-e65e-40ef-83f6-4dd35c3fdf5c-kube-api-access-tcgmc\") pod \"migrator-d57d58cc9-ldjm2\" (UID: \"8801a073-e65e-40ef-83f6-4dd35c3fdf5c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" Jul 06 19:16:03.912314 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:03.912214 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" Jul 06 19:16:04.037706 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.037668 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2"] Jul 06 19:16:04.040466 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:04.040437 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8801a073_e65e_40ef_83f6_4dd35c3fdf5c.slice/crio-8bdf409b364d003d4c5156c7b2f5d62ae3da1916b76c9882fc5a5ec91a6b7a0d WatchSource:0}: Error finding container 8bdf409b364d003d4c5156c7b2f5d62ae3da1916b76c9882fc5a5ec91a6b7a0d: Status 404 returned error can't find the container with id 8bdf409b364d003d4c5156c7b2f5d62ae3da1916b76c9882fc5a5ec91a6b7a0d Jul 06 19:16:04.226578 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.226545 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:04.226764 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.226646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:04.226764 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.226669 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:04.226764 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.226730 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.226714371 +0000 UTC m=+277.756059373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:04.226910 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.226797 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.226910 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.226848 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:06.226836552 +0000 UTC m=+277.756181556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:04.327494 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.327456 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:04.327623 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.327591 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:04.327678 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.327654 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:16:12.32763841 +0000 UTC m=+283.856983413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:04.428653 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.428620 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:16:04.428831 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.428774 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:16:04.428870 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:04.428837 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls podName:79110540-89d9-4f26-a37d-2fa65bcc4922 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:12.42882177 +0000 UTC m=+283.958166776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-29z7q" (UID: "79110540-89d9-4f26-a37d-2fa65bcc4922") : secret "samples-operator-tls" not found Jul 06 19:16:04.440492 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.440470 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7jbm_3b192ffc-504c-4231-9c5a-efa16fdac630/dns-node-resolver/0.log" Jul 06 19:16:04.797155 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:04.797121 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" event={"ID":"8801a073-e65e-40ef-83f6-4dd35c3fdf5c","Type":"ContainerStarted","Data":"8bdf409b364d003d4c5156c7b2f5d62ae3da1916b76c9882fc5a5ec91a6b7a0d"} Jul 06 19:16:05.441769 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:05.441736 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qbrh_3e1ddb84-c2f2-42ce-9438-e2b4bf546381/node-ca/0.log" Jul 06 19:16:05.801975 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:05.801942 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" event={"ID":"8801a073-e65e-40ef-83f6-4dd35c3fdf5c","Type":"ContainerStarted","Data":"6939c437ca88ff06d958424a2eb328a1201b9ae829dd6405d06b340fe99e5adf"} Jul 06 19:16:06.041294 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:06.041210 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zn82_3071cbb5-ad05-492b-a327-1cd14a91e3c4/serve-healthcheck-canary/0.log" Jul 06 19:16:06.245476 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:06.245437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:06.245650 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:06.245513 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:06.245650 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:06.245566 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:06.245650 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:06.245606 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.245650 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:06.245644 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.245627563 +0000 UTC m=+281.774972566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:06.245788 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:06.245659 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:10.245652919 +0000 UTC m=+281.774997923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:06.806467 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:06.806428 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" event={"ID":"8801a073-e65e-40ef-83f6-4dd35c3fdf5c","Type":"ContainerStarted","Data":"a74d97b5be31bb625ade15eb341189244596b599e6a8e4705d45b1bfe40b4eee"} Jul 06 19:16:06.827432 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:06.827320 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-ldjm2" podStartSLOduration=2.152515509 podStartE2EDuration="3.827304372s" totalImagesPullingTime="1.674788863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:06.826590449 +0000 UTC m=+278.355935475" watchObservedRunningTime="2026-07-06 19:16:06.827304372 +0000 UTC m=+278.356649396" Jul 06 19:16:10.277561 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:10.277516 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:10.278006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:10.277605 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:10.278006 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:10.277670 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:16:10.278006 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:10.277688 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:10.278006 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:10.277722 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.277708136 +0000 UTC m=+289.807053140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : configmap "kube-rbac-proxy" not found Jul 06 19:16:10.278006 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:10.277762 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:18.277742334 +0000 UTC m=+289.807087356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:11.517129 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:11.517092 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tckbk" podUID="c0d423a6-8e0c-44a4-a3e2-4a8a76e80633" Jul 06 19:16:11.517129 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:11.517094 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" podUID="7be2f178-730b-4a7e-8a12-a163c0649821" Jul 06 19:16:11.517559 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:11.517109 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" Jul 06 19:16:11.819602 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:11.819528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:11.819759 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:11.819528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:11.819759 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:11.819537 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:16:12.396003 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.395963 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:12.396151 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:12.396105 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:12.396186 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:12.396167 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls podName:3528c111-3fde-4c3a-921c-744cfc1c4ade nodeName:}" failed. No retries permitted until 2026-07-06 19:16:28.396150439 +0000 UTC m=+299.925495443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-rnz9c" (UID: "3528c111-3fde-4c3a-921c-744cfc1c4ade") : secret "cluster-monitoring-operator-tls" not found Jul 06 19:16:12.497293 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.497261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:16:12.499707 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.499683 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/79110540-89d9-4f26-a37d-2fa65bcc4922-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-29z7q\" (UID: \"79110540-89d9-4f26-a37d-2fa65bcc4922\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:16:12.552866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.552841 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" Jul 06 19:16:12.674649 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.674513 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q"] Jul 06 19:16:12.823106 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:12.823078 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" event={"ID":"79110540-89d9-4f26-a37d-2fa65bcc4922","Type":"ContainerStarted","Data":"2ff9f36977bbf614844311153cda786efae036cac71aadb3e09909cc87c4c93f"} Jul 06 19:16:15.322217 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.322190 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:16:15.322629 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.322255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:15.322629 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:15.322303 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:16:15.322629 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:15.322383 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert podName:7be2f178-730b-4a7e-8a12-a163c0649821 nodeName:}" failed. No retries permitted until 2026-07-06 19:18:17.322366801 +0000 UTC m=+408.851711823 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-4rglq" (UID: "7be2f178-730b-4a7e-8a12-a163c0649821") : secret "networking-console-plugin-cert" not found Jul 06 19:16:15.324645 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.324626 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"image-registry-56bc5db5bb-ttfgp\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:15.422580 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.422546 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lwjk9\"" Jul 06 19:16:15.422740 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.422552 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:15.425006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.424986 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0d423a6-8e0c-44a4-a3e2-4a8a76e80633-metrics-tls\") pod \"dns-default-tckbk\" (UID: \"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633\") " pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:15.430994 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.430973 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:15.561863 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.561827 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:16:15.565603 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:15.565580 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda328252d_e23e_4e49_9e96_7f753a7d9bd3.slice/crio-cfe2cce505ca1d9fe7e83180c61a04409785e034150737c13ff78b17a1ce9742 WatchSource:0}: Error finding container cfe2cce505ca1d9fe7e83180c61a04409785e034150737c13ff78b17a1ce9742: Status 404 returned error can't find the container with id cfe2cce505ca1d9fe7e83180c61a04409785e034150737c13ff78b17a1ce9742 Jul 06 19:16:15.722942 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.722911 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8xcq\"" Jul 06 19:16:15.731188 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.731138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:15.835024 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.834990 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" event={"ID":"a328252d-e23e-4e49-9e96-7f753a7d9bd3","Type":"ContainerStarted","Data":"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba"} Jul 06 19:16:15.835199 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.835060 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" event={"ID":"a328252d-e23e-4e49-9e96-7f753a7d9bd3","Type":"ContainerStarted","Data":"cfe2cce505ca1d9fe7e83180c61a04409785e034150737c13ff78b17a1ce9742"} Jul 06 19:16:15.835199 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.835112 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:15.836781 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.836756 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" event={"ID":"79110540-89d9-4f26-a37d-2fa65bcc4922","Type":"ContainerStarted","Data":"2ab3eda27c161c1f2ff70404e0e752731fa8c8321b04f13c3e332f51fb6042fc"} Jul 06 19:16:15.836880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.836800 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" event={"ID":"79110540-89d9-4f26-a37d-2fa65bcc4922","Type":"ContainerStarted","Data":"d092ec6fcac46ebaf0aa06d3e1802c0fc87c1933b8e494ff7de28d6724d40ed1"} Jul 06 19:16:15.857443 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.857417 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tckbk"] Jul 06 19:16:15.859194 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.859145 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" podStartSLOduration=285.85912755 podStartE2EDuration="4m45.85912755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:15.858720719 +0000 UTC m=+287.388065765" watchObservedRunningTime="2026-07-06 19:16:15.85912755 +0000 UTC m=+287.388472576" Jul 06 19:16:15.860603 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:15.860577 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d423a6_8e0c_44a4_a3e2_4a8a76e80633.slice/crio-b6b7489a51ab69856fb7cc0d4acff66a19d2ee549eeeebf8ecd871b5cbd64082 WatchSource:0}: Error finding container b6b7489a51ab69856fb7cc0d4acff66a19d2ee549eeeebf8ecd871b5cbd64082: Status 404 returned error can't find the container with id b6b7489a51ab69856fb7cc0d4acff66a19d2ee549eeeebf8ecd871b5cbd64082 Jul 06 19:16:15.875448 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:15.875412 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-29z7q" podStartSLOduration=17.394663631 podStartE2EDuration="19.875399345s" totalImagesPullingTime="2.480735714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:15.874663809 +0000 UTC m=+287.404008835" watchObservedRunningTime="2026-07-06 19:16:15.875399345 +0000 UTC m=+287.404744348" Jul 06 19:16:16.840531 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:16.840490 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tckbk" event={"ID":"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633","Type":"ContainerStarted","Data":"b6b7489a51ab69856fb7cc0d4acff66a19d2ee549eeeebf8ecd871b5cbd64082"} Jul 06 19:16:18.349747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:18.349712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:18.350193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:18.349781 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:18.350193 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:18.349900 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.350193 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:18.349982 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls podName:60123861-df62-4e28-8192-5ec05132a8d8 nodeName:}" failed. No retries permitted until 2026-07-06 19:16:34.349962016 +0000 UTC m=+305.879307031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fxmgm" (UID: "60123861-df62-4e28-8192-5ec05132a8d8") : secret "insights-runtime-extractor-tls" not found Jul 06 19:16:18.350390 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:18.350259 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60123861-df62-4e28-8192-5ec05132a8d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:19.851043 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:19.851012 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tckbk" event={"ID":"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633","Type":"ContainerStarted","Data":"bcdc0ff139d3d892be88b1bb0f52293cd5e4b57f331e705b38846610fe47b336"} Jul 06 19:16:19.851432 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:19.851062 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tckbk" event={"ID":"c0d423a6-8e0c-44a4-a3e2-4a8a76e80633","Type":"ContainerStarted","Data":"617a3ebf2da19679552efaa69d63e4a1731f93d2af679884e4bf9163c602c5d6"} Jul 06 19:16:19.851432 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:19.851120 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:19.869745 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:19.869697 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tckbk" podStartSLOduration=251.544678429 podStartE2EDuration="4m14.869683543s" totalImagesPullingTime="3.325005114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:19.869453951 +0000 UTC m=+291.398798976" watchObservedRunningTime="2026-07-06 19:16:19.869683543 +0000 UTC m=+291.399028570" Jul 06 19:16:23.012697 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:23.012655 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q7p7g" podUID="c193a56a-2f2c-4eac-86bd-e053d5b7bd57" Jul 06 19:16:27.971090 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.971059 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-knrdz"] Jul 06 19:16:27.976751 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.976734 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:27.978842 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.978821 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:16:27.979145 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.979129 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t2wlc\"" Jul 06 19:16:27.979291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.979278 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:16:27.991092 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:27.991073 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-knrdz"] Jul 06 19:16:28.015754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.015719 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq7dm\" (UniqueName: \"kubernetes.io/projected/b9120fd3-d5a2-46ca-af9f-d75952e2dad2-kube-api-access-wq7dm\") pod \"downloads-685bfb5f64-knrdz\" (UID: \"b9120fd3-d5a2-46ca-af9f-d75952e2dad2\") " pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:28.116944 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.116913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq7dm\" (UniqueName: \"kubernetes.io/projected/b9120fd3-d5a2-46ca-af9f-d75952e2dad2-kube-api-access-wq7dm\") pod \"downloads-685bfb5f64-knrdz\" (UID: \"b9120fd3-d5a2-46ca-af9f-d75952e2dad2\") " pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:28.126551 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.126529 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq7dm\" (UniqueName: \"kubernetes.io/projected/b9120fd3-d5a2-46ca-af9f-d75952e2dad2-kube-api-access-wq7dm\") pod \"downloads-685bfb5f64-knrdz\" (UID: \"b9120fd3-d5a2-46ca-af9f-d75952e2dad2\") " pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:28.286130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.286038 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:28.415984 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.415949 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-knrdz"] Jul 06 19:16:28.418979 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.418958 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:28.419905 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:28.419875 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9120fd3_d5a2_46ca_af9f_d75952e2dad2.slice/crio-704009c7186c78622e62715e7b057f87066e54aa9e414e48479e2a57e106eb7d WatchSource:0}: Error finding container 704009c7186c78622e62715e7b057f87066e54aa9e414e48479e2a57e106eb7d: Status 404 returned error can't find the container with id 704009c7186c78622e62715e7b057f87066e54aa9e414e48479e2a57e106eb7d Jul 06 19:16:28.421645 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.421575 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3528c111-3fde-4c3a-921c-744cfc1c4ade-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-rnz9c\" (UID: \"3528c111-3fde-4c3a-921c-744cfc1c4ade\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:28.651990 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.651898 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" Jul 06 19:16:28.779494 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.779467 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c"] Jul 06 19:16:28.782247 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:28.782224 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3528c111_3fde_4c3a_921c_744cfc1c4ade.slice/crio-4d1235b91c551fa2ab228dfd417d267387fa7e7e77758f33fd94c8bfb84599e0 WatchSource:0}: Error finding container 4d1235b91c551fa2ab228dfd417d267387fa7e7e77758f33fd94c8bfb84599e0: Status 404 returned error can't find the container with id 4d1235b91c551fa2ab228dfd417d267387fa7e7e77758f33fd94c8bfb84599e0 Jul 06 19:16:28.887207 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.887130 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-knrdz" event={"ID":"b9120fd3-d5a2-46ca-af9f-d75952e2dad2","Type":"ContainerStarted","Data":"704009c7186c78622e62715e7b057f87066e54aa9e414e48479e2a57e106eb7d"} Jul 06 19:16:28.888224 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.888193 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" event={"ID":"3528c111-3fde-4c3a-921c-744cfc1c4ade","Type":"ContainerStarted","Data":"4d1235b91c551fa2ab228dfd417d267387fa7e7e77758f33fd94c8bfb84599e0"} Jul 06 19:16:28.893272 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.893246 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:16:28.895386 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.895343 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:16:28.898734 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:28.898710 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:16:29.856670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:29.856643 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tckbk" Jul 06 19:16:31.901187 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.901109 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" event={"ID":"3528c111-3fde-4c3a-921c-744cfc1c4ade","Type":"ContainerStarted","Data":"7d78193a66b548796d07cc88a08d43614aff6245ea817af0699209e1b14106b4"} Jul 06 19:16:31.918293 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.918241 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-rnz9c" podStartSLOduration=33.248275677 podStartE2EDuration="35.918223412s" totalImagesPullingTime="2.669947735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:31.916980182 +0000 UTC m=+303.446325208" watchObservedRunningTime="2026-07-06 19:16:31.918223412 +0000 UTC m=+303.447568436" Jul 06 19:16:31.946068 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.946035 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf"] Jul 06 19:16:31.949183 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.949163 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:31.952523 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.952497 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:16:31.952649 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.952569 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-skw7z\"" Jul 06 19:16:31.959640 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:31.959598 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf"] Jul 06 19:16:32.049163 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.049113 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6vxwf\" (UID: \"b197e777-3e93-4497-a0e6-50e7cd95708e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:32.149486 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.149454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6vxwf\" (UID: \"b197e777-3e93-4497-a0e6-50e7cd95708e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:32.149669 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:32.149566 2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 06 19:16:32.149669 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:32.149617 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates podName:b197e777-3e93-4497-a0e6-50e7cd95708e nodeName:}" failed. No retries permitted until 2026-07-06 19:16:32.649603156 +0000 UTC m=+304.178948159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-6vxwf" (UID: "b197e777-3e93-4497-a0e6-50e7cd95708e") : secret "prometheus-operator-admission-webhook-tls" not found Jul 06 19:16:32.652394 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.652344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6vxwf\" (UID: \"b197e777-3e93-4497-a0e6-50e7cd95708e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:32.654785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.654752 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b197e777-3e93-4497-a0e6-50e7cd95708e-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6vxwf\" (UID: \"b197e777-3e93-4497-a0e6-50e7cd95708e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:32.859998 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.859961 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:32.990513 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.990487 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf"] Jul 06 19:16:32.993419 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:32.993393 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb197e777_3e93_4497_a0e6_50e7cd95708e.slice/crio-0660804daacd159b78f1c179404569d95e342e903f24821f212eab1a8df7e43b WatchSource:0}: Error finding container 0660804daacd159b78f1c179404569d95e342e903f24821f212eab1a8df7e43b: Status 404 returned error can't find the container with id 0660804daacd159b78f1c179404569d95e342e903f24821f212eab1a8df7e43b Jul 06 19:16:32.995838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:32.995819 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:16:33.910941 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:33.910905 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" event={"ID":"b197e777-3e93-4497-a0e6-50e7cd95708e","Type":"ContainerStarted","Data":"0660804daacd159b78f1c179404569d95e342e903f24821f212eab1a8df7e43b"} Jul 06 19:16:34.365270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.365233 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:34.368233 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.368187 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60123861-df62-4e28-8192-5ec05132a8d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fxmgm\" (UID: \"60123861-df62-4e28-8192-5ec05132a8d8\") " pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:34.552655 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.552626 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2n6g9\"" Jul 06 19:16:34.560472 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.560453 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fxmgm" Jul 06 19:16:34.715453 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.715269 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fxmgm"] Jul 06 19:16:34.813091 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:34.813060 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60123861_df62_4e28_8192_5ec05132a8d8.slice/crio-cdcf3657b8384bb982179d29b79d4a18b021125d839db21ddc9927265b8bba01 WatchSource:0}: Error finding container cdcf3657b8384bb982179d29b79d4a18b021125d839db21ddc9927265b8bba01: Status 404 returned error can't find the container with id cdcf3657b8384bb982179d29b79d4a18b021125d839db21ddc9927265b8bba01 Jul 06 19:16:34.916630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.916595 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" event={"ID":"b197e777-3e93-4497-a0e6-50e7cd95708e","Type":"ContainerStarted","Data":"0e498c850f962ca102978b1b8615b1f09c3b979d1fab8d20edca4fa451a7fe69"} Jul 06 19:16:34.916804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.916777 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:34.917814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.917787 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fxmgm" event={"ID":"60123861-df62-4e28-8192-5ec05132a8d8","Type":"ContainerStarted","Data":"cdcf3657b8384bb982179d29b79d4a18b021125d839db21ddc9927265b8bba01"} Jul 06 19:16:34.921986 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.921959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" Jul 06 19:16:34.933413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:34.933371 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6vxwf" podStartSLOduration=2.102317508 podStartE2EDuration="3.93333333s" totalImagesPullingTime="1.831015822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:34.932900333 +0000 UTC m=+306.462245358" watchObservedRunningTime="2026-07-06 19:16:34.93333333 +0000 UTC m=+306.462678354" Jul 06 19:16:35.439007 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:35.438966 2558 patch_prober.go:28] interesting pod/image-registry-56bc5db5bb-ttfgp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:16:35.439493 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:35.439036 2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:16:36.014961 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.014926 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd"] Jul 06 19:16:36.018410 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.018384 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.021804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.021565 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 19:16:36.021804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.021570 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 19:16:36.021804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.021626 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlkcm\"" Jul 06 19:16:36.022265 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.022247 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:16:36.039168 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.039139 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd"] Jul 06 19:16:36.080288 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.080251 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhtn\" (UniqueName: \"kubernetes.io/projected/b373003a-544d-466f-a966-37728caa365f-kube-api-access-qqhtn\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.080506 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.080316 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.080506 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.080410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.080617 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.080503 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b373003a-544d-466f-a966-37728caa365f-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.181922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.181886 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b373003a-544d-466f-a966-37728caa365f-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.181922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.181924 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhtn\" (UniqueName: \"kubernetes.io/projected/b373003a-544d-466f-a966-37728caa365f-kube-api-access-qqhtn\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.182179 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.181984 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.182179 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.182060 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.182833 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.182808 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b373003a-544d-466f-a966-37728caa365f-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.185453 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.185410 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.185453 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.185426 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b373003a-544d-466f-a966-37728caa365f-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.192154 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.192106 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhtn\" (UniqueName: \"kubernetes.io/projected/b373003a-544d-466f-a966-37728caa365f-kube-api-access-qqhtn\") pod \"prometheus-operator-567f7b8b7c-7sthd\" (UID: \"b373003a-544d-466f-a966-37728caa365f\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.330615 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.330528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" Jul 06 19:16:36.482064 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.482017 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd"] Jul 06 19:16:36.770922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.770891 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:16:36.781540 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.781509 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.791731 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.791705 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:16:36.791869 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.791756 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8p8r\"" Jul 06 19:16:36.791869 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.791787 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:16:36.792262 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.792243 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:16:36.792380 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.792322 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:16:36.795925 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.795900 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:16:36.800395 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.800344 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:16:36.847725 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.847698 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:16:36.889774 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889433 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.889774 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889498 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.889774 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.889774 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889710 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtrx\" (UniqueName: \"kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.890143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889827 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.890143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.889860 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990297 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990499 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990334 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtrx\" (UniqueName: \"kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990634 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990601 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990766 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990766 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.990766 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.990760 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.991496 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.991445 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.991496 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.991445 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.991658 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.991494 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.993684 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.993652 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:36.993948 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:36.993926 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:37.000150 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:37.000126 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtrx\" (UniqueName: \"kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx\") pod \"console-7d45d6d5c7-c4cdp\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:37.094217 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:37.094136 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:37.811802 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:37.811770 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb373003a_544d_466f_a966_37728caa365f.slice/crio-bdb9894e8efbb1436013da6eaab995df67e3594903ef6629e1eb735621878654 WatchSource:0}: Error finding container bdb9894e8efbb1436013da6eaab995df67e3594903ef6629e1eb735621878654: Status 404 returned error can't find the container with id bdb9894e8efbb1436013da6eaab995df67e3594903ef6629e1eb735621878654 Jul 06 19:16:37.931177 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:37.931129 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" event={"ID":"b373003a-544d-466f-a966-37728caa365f","Type":"ContainerStarted","Data":"bdb9894e8efbb1436013da6eaab995df67e3594903ef6629e1eb735621878654"} Jul 06 19:16:38.011999 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:38.011955 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:16:38.903486 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:38.903459 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:16:38.906422 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:38.906393 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220568ce_7333_4f1d_80ad_bd2a625fd049.slice/crio-b2171b9650723df2d108cb869ce389bacddc1ba95e231dfa2e71e254407db23c WatchSource:0}: Error finding container b2171b9650723df2d108cb869ce389bacddc1ba95e231dfa2e71e254407db23c: Status 404 returned error can't find the container with id b2171b9650723df2d108cb869ce389bacddc1ba95e231dfa2e71e254407db23c Jul 06 19:16:38.936043 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:38.936009 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d6d5c7-c4cdp" event={"ID":"220568ce-7333-4f1d-80ad-bd2a625fd049","Type":"ContainerStarted","Data":"b2171b9650723df2d108cb869ce389bacddc1ba95e231dfa2e71e254407db23c"} Jul 06 19:16:38.937228 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:38.937204 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fxmgm" event={"ID":"60123861-df62-4e28-8192-5ec05132a8d8","Type":"ContainerStarted","Data":"a56790099fc55ef5cd6643dd35e58f23bbe2f87f2b424f2a2c4b4f35e19da330"} Jul 06 19:16:39.944443 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:39.944404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fxmgm" event={"ID":"60123861-df62-4e28-8192-5ec05132a8d8","Type":"ContainerStarted","Data":"71dcf9afc75b68653e42a5f22e6e44e50951699d771b4b9e8652c0dfd64b3e0b"} Jul 06 19:16:40.950479 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:40.950434 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" event={"ID":"b373003a-544d-466f-a966-37728caa365f","Type":"ContainerStarted","Data":"3db0a53c4bac88f61ec9dd685c45774a800741975f86cdcdb49d123458245957"} Jul 06 19:16:40.950875 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:40.950497 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" event={"ID":"b373003a-544d-466f-a966-37728caa365f","Type":"ContainerStarted","Data":"f694f77004a65ba7b42fb463fb406571a3382a359587252f621633cb8e339d58"} Jul 06 19:16:42.404702 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.404447 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-7sthd" podStartSLOduration=5.093402212 podStartE2EDuration="7.404422362s" totalImagesPullingTime="2.31102015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:40.968460606 +0000 UTC m=+312.497805653" watchObservedRunningTime="2026-07-06 19:16:42.404422362 +0000 UTC m=+313.933767392" Jul 06 19:16:42.406160 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.406114 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg"] Jul 06 19:16:42.417307 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.417285 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.420093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.420066 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:16:42.420221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.420180 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-58dlr\"" Jul 06 19:16:42.420299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.420283 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:16:42.424374 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.424218 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg"] Jul 06 19:16:42.459475 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.459440 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sjgpq"] Jul 06 19:16:42.464711 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.464683 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.468670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.468532 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:16:42.468670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.468536 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:16:42.468670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.468616 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l478r\"" Jul 06 19:16:42.468958 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.468926 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:16:42.541689 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541657 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7mq\" (UniqueName: \"kubernetes.io/projected/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-kube-api-access-bk7mq\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541689 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541705 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-root\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541731 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-metrics-client-ca\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541801 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dbd6613-0437-4f52-b622-9b3ae215edec-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541831 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-wtmp\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541874 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-textfile\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541893 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.541931 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541913 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.542252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541949 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.542252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.541976 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.542252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.542022 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-sys\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.542252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.542054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plfrl\" (UniqueName: \"kubernetes.io/projected/8dbd6613-0437-4f52-b622-9b3ae215edec-kube-api-access-plfrl\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.542252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.542073 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-tls\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-sys\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plfrl\" (UniqueName: \"kubernetes.io/projected/8dbd6613-0437-4f52-b622-9b3ae215edec-kube-api-access-plfrl\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643115 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-tls\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643153 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-sys\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7mq\" (UniqueName: \"kubernetes.io/projected/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-kube-api-access-bk7mq\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643260 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-root\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-metrics-client-ca\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643303 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dbd6613-0437-4f52-b622-9b3ae215edec-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.643338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-wtmp\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643395 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-root\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643482 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-wtmp\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643518 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-textfile\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643554 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643588 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.643630 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643619 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643902 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643653 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.643902 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:42.643763 2558 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 19:16:42.643902 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643800 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-textfile\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.643902 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:16:42.643819 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls podName:8dbd6613-0437-4f52-b622-9b3ae215edec nodeName:}" failed. No retries permitted until 2026-07-06 19:16:43.143801436 +0000 UTC m=+314.673146438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-7vshg" (UID: "8dbd6613-0437-4f52-b622-9b3ae215edec") : secret "openshift-state-metrics-tls" not found Jul 06 19:16:42.644113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.643948 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-metrics-client-ca\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.644113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.644096 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dbd6613-0437-4f52-b622-9b3ae215edec-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.644274 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.644252 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-accelerators-collector-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.646280 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.646238 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.646628 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.646582 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.647031 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.647009 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-node-exporter-tls\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.652503 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.652388 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7mq\" (UniqueName: \"kubernetes.io/projected/4725c635-a8dc-41b7-8ddd-ee75b0e25ba6-kube-api-access-bk7mq\") pod \"node-exporter-sjgpq\" (UID: \"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6\") " pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:42.652503 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.652465 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plfrl\" (UniqueName: \"kubernetes.io/projected/8dbd6613-0437-4f52-b622-9b3ae215edec-kube-api-access-plfrl\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:42.775935 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:42.775903 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sjgpq" Jul 06 19:16:43.099714 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.099629 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:16:43.102969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.102944 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.110745 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.110706 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:16:43.114549 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.114524 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:16:43.148182 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.148152 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:43.151231 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.151207 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8dbd6613-0437-4f52-b622-9b3ae215edec-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7vshg\" (UID: \"8dbd6613-0437-4f52-b622-9b3ae215edec\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:43.249015 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.248972 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249177 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249107 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249239 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249310 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249233 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249310 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249310 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6phh\" (UniqueName: \"kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.249443 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.249390 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.330225 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.329998 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350731 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350781 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6phh\" (UniqueName: \"kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.350821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.352286 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.353422 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.352571 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.354560 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.354004 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.354560 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.354515 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.360876 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.360849 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.363300 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.363253 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6phh\" (UniqueName: \"kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.363523 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.363500 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config\") pod \"console-dc7d88b4d-7vvtb\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.414549 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.414519 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:43.913502 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:43.913470 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4725c635_a8dc_41b7_8ddd_ee75b0e25ba6.slice/crio-732c0ddc7940875e11d06908503de670555a85c3e82d67bb5aa4f7e1d93b14bf WatchSource:0}: Error finding container 732c0ddc7940875e11d06908503de670555a85c3e82d67bb5aa4f7e1d93b14bf: Status 404 returned error can't find the container with id 732c0ddc7940875e11d06908503de670555a85c3e82d67bb5aa4f7e1d93b14bf Jul 06 19:16:43.972273 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:43.972190 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjgpq" event={"ID":"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6","Type":"ContainerStarted","Data":"732c0ddc7940875e11d06908503de670555a85c3e82d67bb5aa4f7e1d93b14bf"} Jul 06 19:16:44.088974 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.088940 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg"] Jul 06 19:16:44.092244 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:44.092218 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbd6613_0437_4f52_b622_9b3ae215edec.slice/crio-fc9ed26a39739b33bf474bb629de64997d5533d65a4d2aac339291c04becc343 WatchSource:0}: Error finding container fc9ed26a39739b33bf474bb629de64997d5533d65a4d2aac339291c04becc343: Status 404 returned error can't find the container with id fc9ed26a39739b33bf474bb629de64997d5533d65a4d2aac339291c04becc343 Jul 06 19:16:44.107836 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.107812 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:16:44.111009 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:44.110979 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eccb27_4d22_4b06_9b8f_fac9593c979f.slice/crio-d2b8c39e07e90b18086cc752f182e1ea90b7f3d689fb7c7f8d34063e8d239e0c WatchSource:0}: Error finding container d2b8c39e07e90b18086cc752f182e1ea90b7f3d689fb7c7f8d34063e8d239e0c: Status 404 returned error can't find the container with id d2b8c39e07e90b18086cc752f182e1ea90b7f3d689fb7c7f8d34063e8d239e0c Jul 06 19:16:44.979595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.979137 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d6d5c7-c4cdp" event={"ID":"220568ce-7333-4f1d-80ad-bd2a625fd049","Type":"ContainerStarted","Data":"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039"} Jul 06 19:16:44.981459 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.981429 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjgpq" event={"ID":"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6","Type":"ContainerStarted","Data":"bb23e41870e90330b39a818fcceee55f554115552a3b6a2f81663940851a4c00"} Jul 06 19:16:44.983822 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.983756 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" event={"ID":"8dbd6613-0437-4f52-b622-9b3ae215edec","Type":"ContainerStarted","Data":"286ab7e9cf29dbe46282b7caaacb1e806bf1c1753397a977e6d949410047ed31"} Jul 06 19:16:44.984061 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.983960 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" event={"ID":"8dbd6613-0437-4f52-b622-9b3ae215edec","Type":"ContainerStarted","Data":"f7a3c8eca1027bd2ff91fc8c84a2494aa9cefb9dfd12ef8e84b2582edb9a4ad8"} Jul 06 19:16:44.984061 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.984027 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" event={"ID":"8dbd6613-0437-4f52-b622-9b3ae215edec","Type":"ContainerStarted","Data":"fc9ed26a39739b33bf474bb629de64997d5533d65a4d2aac339291c04becc343"} Jul 06 19:16:44.985780 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.985720 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc7d88b4d-7vvtb" event={"ID":"c6eccb27-4d22-4b06-9b8f-fac9593c979f","Type":"ContainerStarted","Data":"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313"} Jul 06 19:16:44.985780 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.985755 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc7d88b4d-7vvtb" event={"ID":"c6eccb27-4d22-4b06-9b8f-fac9593c979f","Type":"ContainerStarted","Data":"d2b8c39e07e90b18086cc752f182e1ea90b7f3d689fb7c7f8d34063e8d239e0c"} Jul 06 19:16:44.988838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:44.988646 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fxmgm" event={"ID":"60123861-df62-4e28-8192-5ec05132a8d8","Type":"ContainerStarted","Data":"d16402c0cd126b2071cecea6aaede36c7522ebca161f30e627e145683b6f35ba"} Jul 06 19:16:45.014239 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.014183 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d45d6d5c7-c4cdp" podStartSLOduration=4.003321067 podStartE2EDuration="9.014160979s" totalImagesPullingTime="5.010839912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:45.014161504 +0000 UTC m=+316.543506530" watchObservedRunningTime="2026-07-06 19:16:45.014160979 +0000 UTC m=+316.543506015" Jul 06 19:16:45.036894 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.036835 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fxmgm" podStartSLOduration=34.07680138 podStartE2EDuration="43.036812779s" totalImagesPullingTime="8.960011399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:45.035401173 +0000 UTC m=+316.564746199" watchObservedRunningTime="2026-07-06 19:16:45.036812779 +0000 UTC m=+316.566157807" Jul 06 19:16:45.086228 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.086184 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc7d88b4d-7vvtb" podStartSLOduration=2.086170081 podStartE2EDuration="2.086170081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:45.08479625 +0000 UTC m=+316.614141275" watchObservedRunningTime="2026-07-06 19:16:45.086170081 +0000 UTC m=+316.615515084" Jul 06 19:16:45.469828 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.469791 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54546fdfc-tpwf5"] Jul 06 19:16:45.474168 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.474143 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.476558 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.476535 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 19:16:45.477149 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477128 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 19:16:45.477296 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477279 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c8d5e90lektu9\"" Jul 06 19:16:45.477421 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477307 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 19:16:45.477421 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477333 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tjzg5\"" Jul 06 19:16:45.477529 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477140 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 19:16:45.477529 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.477146 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 19:16:45.487522 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.487500 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54546fdfc-tpwf5"] Jul 06 19:16:45.573711 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573679 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.573853 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.573853 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.573957 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573861 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxcm\" (UniqueName: \"kubernetes.io/projected/57605514-10be-447f-ac0b-00b057799d50-kube-api-access-vtxcm\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.573957 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573898 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57605514-10be-447f-ac0b-00b057799d50-metrics-client-ca\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.574047 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.573982 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.574047 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.574021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.574123 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.574055 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-grpc-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.674969 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.674932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675167 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.674984 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675167 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675059 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675167 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675128 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxcm\" (UniqueName: \"kubernetes.io/projected/57605514-10be-447f-ac0b-00b057799d50-kube-api-access-vtxcm\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675167 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675150 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57605514-10be-447f-ac0b-00b057799d50-metrics-client-ca\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675195 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675224 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.675399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.675250 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-grpc-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.676060 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.676015 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57605514-10be-447f-ac0b-00b057799d50-metrics-client-ca\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.678482 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.678434 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.678667 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.678642 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.679023 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.679001 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.679100 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.679058 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.679368 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.679329 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-grpc-tls\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.679470 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.679445 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57605514-10be-447f-ac0b-00b057799d50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.687714 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.687671 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxcm\" (UniqueName: \"kubernetes.io/projected/57605514-10be-447f-ac0b-00b057799d50-kube-api-access-vtxcm\") pod \"thanos-querier-54546fdfc-tpwf5\" (UID: \"57605514-10be-447f-ac0b-00b057799d50\") " pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.786755 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.786673 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:16:45.948746 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.948713 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54546fdfc-tpwf5"] Jul 06 19:16:45.952446 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:45.952421 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57605514_10be_447f_ac0b_00b057799d50.slice/crio-914db7950efab63114c5793dfff0ab7c4202c50b53795f38855c2794ea455686 WatchSource:0}: Error finding container 914db7950efab63114c5793dfff0ab7c4202c50b53795f38855c2794ea455686: Status 404 returned error can't find the container with id 914db7950efab63114c5793dfff0ab7c4202c50b53795f38855c2794ea455686 Jul 06 19:16:45.993896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.993871 2558 generic.go:356] "Generic (PLEG): container finished" podID="4725c635-a8dc-41b7-8ddd-ee75b0e25ba6" containerID="bb23e41870e90330b39a818fcceee55f554115552a3b6a2f81663940851a4c00" exitCode=0 Jul 06 19:16:45.994239 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.993934 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjgpq" event={"ID":"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6","Type":"ContainerDied","Data":"bb23e41870e90330b39a818fcceee55f554115552a3b6a2f81663940851a4c00"} Jul 06 19:16:45.996000 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.995976 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" event={"ID":"8dbd6613-0437-4f52-b622-9b3ae215edec","Type":"ContainerStarted","Data":"32b429fddcb0dde9413eef5d10c937d75752a69fc7af4f1a154f99f4c6bc11fd"} Jul 06 19:16:45.998218 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:45.998182 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"914db7950efab63114c5793dfff0ab7c4202c50b53795f38855c2794ea455686"} Jul 06 19:16:46.037718 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.037630 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7vshg" podStartSLOduration=2.4706492349999998 podStartE2EDuration="4.037611007s" totalImagesPullingTime="1.566961772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:46.036782873 +0000 UTC m=+317.566127898" watchObservedRunningTime="2026-07-06 19:16:46.037611007 +0000 UTC m=+317.566956066" Jul 06 19:16:46.809402 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.809368 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-546989849d-wvpt5"] Jul 06 19:16:46.812574 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.812557 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.815799 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.815777 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:16:46.816817 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.816801 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7hnsrggbhirnl\"" Jul 06 19:16:46.817165 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.817149 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:16:46.817239 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.817151 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s5j9c\"" Jul 06 19:16:46.817291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.817216 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:16:46.817291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.817275 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:16:46.823699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.823676 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546989849d-wvpt5"] Jul 06 19:16:46.887747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.887712 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-metrics-server-audit-profiles\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.887968 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.887770 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflgz\" (UniqueName: \"kubernetes.io/projected/4e12011d-76e5-4afe-8b98-62c13720630f-kube-api-access-mflgz\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.887968 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.887809 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-client-certs\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.887968 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.887837 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-client-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.887968 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.887939 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4e12011d-76e5-4afe-8b98-62c13720630f-audit-log\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.888184 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.888008 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.888184 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.888044 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-tls\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989157 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-metrics-server-audit-profiles\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989221 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflgz\" (UniqueName: \"kubernetes.io/projected/4e12011d-76e5-4afe-8b98-62c13720630f-kube-api-access-mflgz\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989254 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-client-certs\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-client-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4e12011d-76e5-4afe-8b98-62c13720630f-audit-log\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989643 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989643 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989452 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-tls\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.989932 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.989898 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4e12011d-76e5-4afe-8b98-62c13720630f-audit-log\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.990504 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.990453 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-metrics-server-audit-profiles\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.990607 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.990586 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e12011d-76e5-4afe-8b98-62c13720630f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.992392 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.992330 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-client-ca-bundle\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.992477 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.992444 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-tls\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.992748 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.992706 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4e12011d-76e5-4afe-8b98-62c13720630f-secret-metrics-server-client-certs\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:46.999237 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:46.999209 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflgz\" (UniqueName: \"kubernetes.io/projected/4e12011d-76e5-4afe-8b98-62c13720630f-kube-api-access-mflgz\") pod \"metrics-server-546989849d-wvpt5\" (UID: \"4e12011d-76e5-4afe-8b98-62c13720630f\") " pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:47.006718 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.006686 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjgpq" event={"ID":"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6","Type":"ContainerStarted","Data":"6afb888ba0b72697e0957e89e76fe7523df8c210fb7d0585e7d2c0797d3510a2"} Jul 06 19:16:47.006835 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.006751 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sjgpq" event={"ID":"4725c635-a8dc-41b7-8ddd-ee75b0e25ba6","Type":"ContainerStarted","Data":"9ddd43b206045cd842e1922f2eb777bde21b857a27302505b04bf6c4b2b4ba9f"} Jul 06 19:16:47.030343 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.030288 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sjgpq" podStartSLOduration=3.996476399 podStartE2EDuration="5.030271414s" totalImagesPullingTime="949.603893ms" totalInitContainerRuntime="84.191122ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:47.029442328 +0000 UTC m=+318.558787377" watchObservedRunningTime="2026-07-06 19:16:47.030271414 +0000 UTC m=+318.559616438" Jul 06 19:16:47.095291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.095201 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:47.095291 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.095263 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:47.099869 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.099849 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:47.124438 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.123564 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:16:47.303055 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:47.303022 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-546989849d-wvpt5"] Jul 06 19:16:47.305880 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:47.305853 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e12011d_76e5_4afe_8b98_62c13720630f.slice/crio-981ffd43d071865bbe95d50482da509b188fe01e98e982db1ab1897a115c8bea WatchSource:0}: Error finding container 981ffd43d071865bbe95d50482da509b188fe01e98e982db1ab1897a115c8bea: Status 404 returned error can't find the container with id 981ffd43d071865bbe95d50482da509b188fe01e98e982db1ab1897a115c8bea Jul 06 19:16:48.013005 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.012968 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" event={"ID":"4e12011d-76e5-4afe-8b98-62c13720630f","Type":"ContainerStarted","Data":"981ffd43d071865bbe95d50482da509b188fe01e98e982db1ab1897a115c8bea"} Jul 06 19:16:48.018060 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.018027 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:16:48.320778 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.320695 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:16:48.769928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.769894 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:16:48.784562 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.784529 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.788209 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.787716 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:16:48.789171 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789145 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:16:48.789670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789513 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:16:48.789670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789567 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:16:48.789840 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789669 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:16:48.789840 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789726 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtf9d\"" Jul 06 19:16:48.789840 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789746 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:16:48.789840 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789774 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:16:48.789840 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789815 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:16:48.790143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789917 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:16:48.790143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789969 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:16:48.790143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.789972 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:16:48.790143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.790024 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:16:48.790143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.790127 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:16:48.790707 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.790325 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:16:48.790707 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.790398 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ak0os96me7ume\"" Jul 06 19:16:48.793669 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.793394 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:16:48.912023 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.911987 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912186 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912038 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912186 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912184 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfp4x\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912229 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912254 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912482 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912371 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912482 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912424 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912482 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912453 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912482 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912476 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912499 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912523 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912550 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912588 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912619 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912641 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.912679 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912667 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.913026 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912690 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.913026 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:48.913026 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:48.912764 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.013593 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013551 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013639 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013664 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013686 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013711 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013733 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013766 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013847 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013885 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013931 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfp4x\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.013990 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014062 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014101 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014125 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014149 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014172 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.014962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.014198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.016973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.015876 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.016973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.016018 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.020670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.017392 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.020670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.020125 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.020670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.020393 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.021451 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.022151 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.022655 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.023166 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.024083 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.024669 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.025343 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.025430 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.026619 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.026959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.027875 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.032168 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.032142 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.032425 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.032302 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.032807 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.032770 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.033549 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.033490 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfp4x\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.034251 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.034216 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.102553 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.102198 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:16:49.616265 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:49.616179 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:16:49.619790 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:16:49.619726 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode986178d_1ba0_4d95_b652_4d911b37cf11.slice/crio-504b8033c62a39f9a67679657a4dbf52efdb2af2dbbe6b9d3ae13679b4350520 WatchSource:0}: Error finding container 504b8033c62a39f9a67679657a4dbf52efdb2af2dbbe6b9d3ae13679b4350520: Status 404 returned error can't find the container with id 504b8033c62a39f9a67679657a4dbf52efdb2af2dbbe6b9d3ae13679b4350520 Jul 06 19:16:50.032638 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.032554 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"504b8033c62a39f9a67679657a4dbf52efdb2af2dbbe6b9d3ae13679b4350520"} Jul 06 19:16:50.034636 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.034602 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" event={"ID":"4e12011d-76e5-4afe-8b98-62c13720630f","Type":"ContainerStarted","Data":"c5e89f0ffdfbdf46de25d0a1dba60cf1082ea22ff12fb6b7d6a88501febd0649"} Jul 06 19:16:50.037101 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.037070 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"41ce9faaa5d033389d91df2d2db9229ac1b64817f1b1ec99aa220b8320fc8080"} Jul 06 19:16:50.037218 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.037134 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"98d2ec45160d5c18dad57ee7981d9375a5e29610bd8b4cf4b5ecec3956278ea7"} Jul 06 19:16:50.037218 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.037174 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"6ae3229fe1387d28bd953167302f2f2a8c1b5db1f09941063a7909b08f6e0dae"} Jul 06 19:16:50.056550 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.056501 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" podStartSLOduration=1.912653776 podStartE2EDuration="4.0564865s" totalImagesPullingTime="2.143832724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:50.054951328 +0000 UTC m=+321.584296353" watchObservedRunningTime="2026-07-06 19:16:50.0564865 +0000 UTC m=+321.585831525" Jul 06 19:16:50.061932 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:50.061910 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:16:53.415983 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:53.415938 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:16:58.076415 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:58.076372 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-knrdz" event={"ID":"b9120fd3-d5a2-46ca-af9f-d75952e2dad2","Type":"ContainerStarted","Data":"654074d7ea01ffa847695cca851118c4fe4534d6ce2562c0b384700ca061b7bb"} Jul 06 19:16:58.076890 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:58.076560 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:58.095426 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:58.095393 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-knrdz" Jul 06 19:16:58.096268 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:58.096211 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-knrdz" podStartSLOduration=2.131419938 podStartE2EDuration="31.096192436s" totalImagesPullingTime="28.964772498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:58.094217718 +0000 UTC m=+329.623562762" watchObservedRunningTime="2026-07-06 19:16:58.096192436 +0000 UTC m=+329.625537462" Jul 06 19:16:59.083144 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:59.083111 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" exitCode=0 Jul 06 19:16:59.083592 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:59.083199 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} Jul 06 19:16:59.087117 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:59.087086 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"49616b4f5f180ca00e987b25da15f0eeef50432d1831432499edf8d414489009"} Jul 06 19:16:59.087256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:59.087165 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"3f352ae635db107130bd688f8d4fe5100fd5b5356aa319468ae9175d32b9b875"} Jul 06 19:16:59.192476 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:16:59.192437 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:17:00.099148 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:00.099067 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" event={"ID":"57605514-10be-447f-ac0b-00b057799d50","Type":"ContainerStarted","Data":"414eb61814342489c66e9e21fcddf4feaa7a2fd1323116bde425fee7644f15e2"} Jul 06 19:17:00.099922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:00.099859 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:17:00.108486 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:00.108414 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" Jul 06 19:17:00.127559 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:00.126976 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54546fdfc-tpwf5" podStartSLOduration=2.79397756 podStartE2EDuration="15.126959544s" totalImagesPullingTime="12.332981984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:00.125784437 +0000 UTC m=+331.655129466" watchObservedRunningTime="2026-07-06 19:17:00.126959544 +0000 UTC m=+331.656304569" Jul 06 19:17:05.145764 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:05.145607 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} Jul 06 19:17:05.145764 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:05.145699 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} Jul 06 19:17:06.154137 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:06.154093 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} Jul 06 19:17:06.154620 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:06.154188 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} Jul 06 19:17:06.154620 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:06.154216 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} Jul 06 19:17:06.154620 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:06.154257 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerStarted","Data":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} Jul 06 19:17:06.192337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:06.191315 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.110359504 podStartE2EDuration="18.191287132s" totalImagesPullingTime="15.043282222s" totalInitContainerRuntime="37.645406ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:06.184411242 +0000 UTC m=+337.713756270" watchObservedRunningTime="2026-07-06 19:17:06.191287132 +0000 UTC m=+337.720632160" Jul 06 19:17:07.124158 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:07.124123 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:17:07.124328 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:07.124181 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:17:09.102457 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:09.102416 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:17:13.348751 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.348679 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-dc7d88b4d-7vvtb" podUID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" containerName="console" containerID="cri-o://b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313" gracePeriod=15 Jul 06 19:17:13.632208 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.632183 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:17:13.764899 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.764864 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.764937 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.764966 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6phh\" (UniqueName: \"kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.764983 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765016 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765038 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765073 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config\") pod \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\" (UID: \"c6eccb27-4d22-4b06-9b8f-fac9593c979f\") " Jul 06 19:17:13.765465 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765318 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:13.765465 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765446 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:13.765653 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765611 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:13.765736 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.765717 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config" (OuterVolumeSpecName: "console-config") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:13.767657 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.767637 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:13.767853 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.767833 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:13.767853 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.767844 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh" (OuterVolumeSpecName: "kube-api-access-h6phh") pod "c6eccb27-4d22-4b06-9b8f-fac9593c979f" (UID: "c6eccb27-4d22-4b06-9b8f-fac9593c979f"). InnerVolumeSpecName "kube-api-access-h6phh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866243 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-serving-cert\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866267 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6phh\" (UniqueName: \"kubernetes.io/projected/c6eccb27-4d22-4b06-9b8f-fac9593c979f-kube-api-access-h6phh\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866277 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-service-ca\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866286 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-oauth-serving-cert\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866294 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866302 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6eccb27-4d22-4b06-9b8f-fac9593c979f-console-oauth-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:13.866321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:13.866310 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6eccb27-4d22-4b06-9b8f-fac9593c979f-trusted-ca-bundle\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:14.186496 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.186465 2558 generic.go:356] "Generic (PLEG): container finished" podID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" containerID="b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313" exitCode=0 Jul 06 19:17:14.186675 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.186506 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc7d88b4d-7vvtb" Jul 06 19:17:14.186675 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.186559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc7d88b4d-7vvtb" event={"ID":"c6eccb27-4d22-4b06-9b8f-fac9593c979f","Type":"ContainerDied","Data":"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313"} Jul 06 19:17:14.186675 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.186616 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc7d88b4d-7vvtb" event={"ID":"c6eccb27-4d22-4b06-9b8f-fac9593c979f","Type":"ContainerDied","Data":"d2b8c39e07e90b18086cc752f182e1ea90b7f3d689fb7c7f8d34063e8d239e0c"} Jul 06 19:17:14.186675 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.186645 2558 scope.go:117] "RemoveContainer" containerID="b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313" Jul 06 19:17:14.195825 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.195810 2558 scope.go:117] "RemoveContainer" containerID="b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313" Jul 06 19:17:14.196081 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:17:14.196062 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313\": container with ID starting with b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313 not found: ID does not exist" containerID="b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313" Jul 06 19:17:14.196125 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.196089 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313"} err="failed to get container status \"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313\": rpc error: code = NotFound desc = could not find container \"b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313\": container with ID starting with b697747ac64dcfad67770cfde61b834e1ac1ebadc0eb1490fb37a5b4d6ec4313 not found: ID does not exist" Jul 06 19:17:14.206461 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.206440 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:17:14.208826 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:14.208804 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc7d88b4d-7vvtb"] Jul 06 19:17:15.017106 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.017074 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" path="/var/lib/kubelet/pods/c6eccb27-4d22-4b06-9b8f-fac9593c979f/volumes" Jul 06 19:17:15.085775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.085745 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerName="registry" containerID="cri-o://a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba" gracePeriod=30 Jul 06 19:17:15.355750 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.355726 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:17:15.481302 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481527 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481430 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481527 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481469 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481527 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481506 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhdmz\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481615 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481654 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481682 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted\") pod \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\" (UID: \"a328252d-e23e-4e49-9e96-7f753a7d9bd3\") " Jul 06 19:17:15.481878 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.481739 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:15.482060 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.482037 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-trusted-ca\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.482376 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.482313 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:15.484242 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.484204 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:15.484242 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.484206 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:15.484425 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.484304 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:15.484495 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.484466 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:15.484537 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.484519 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz" (OuterVolumeSpecName: "kube-api-access-zhdmz") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "kube-api-access-zhdmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:15.490492 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.490470 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a328252d-e23e-4e49-9e96-7f753a7d9bd3" (UID: "a328252d-e23e-4e49-9e96-7f753a7d9bd3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583189 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-tls\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583215 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-bound-sa-token\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583224 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-installation-pull-secrets\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583234 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhdmz\" (UniqueName: \"kubernetes.io/projected/a328252d-e23e-4e49-9e96-7f753a7d9bd3-kube-api-access-zhdmz\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583243 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a328252d-e23e-4e49-9e96-7f753a7d9bd3-image-registry-private-configuration\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583253 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a328252d-e23e-4e49-9e96-7f753a7d9bd3-registry-certificates\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:15.583270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:15.583260 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a328252d-e23e-4e49-9e96-7f753a7d9bd3-ca-trust-extracted\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:16.196048 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.196014 2558 generic.go:356] "Generic (PLEG): container finished" podID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerID="a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba" exitCode=0 Jul 06 19:17:16.196571 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.196055 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" event={"ID":"a328252d-e23e-4e49-9e96-7f753a7d9bd3","Type":"ContainerDied","Data":"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba"} Jul 06 19:17:16.196571 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.196061 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" Jul 06 19:17:16.196571 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.196099 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56bc5db5bb-ttfgp" event={"ID":"a328252d-e23e-4e49-9e96-7f753a7d9bd3","Type":"ContainerDied","Data":"cfe2cce505ca1d9fe7e83180c61a04409785e034150737c13ff78b17a1ce9742"} Jul 06 19:17:16.196571 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.196119 2558 scope.go:117] "RemoveContainer" containerID="a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba" Jul 06 19:17:16.204940 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.204922 2558 scope.go:117] "RemoveContainer" containerID="a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba" Jul 06 19:17:16.205181 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:17:16.205162 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba\": container with ID starting with a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba not found: ID does not exist" containerID="a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba" Jul 06 19:17:16.205233 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.205190 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba"} err="failed to get container status \"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba\": rpc error: code = NotFound desc = could not find container \"a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba\": container with ID starting with a26b4d3d673621a3c6b57ab162108dd59bc4de93c4af4b1f32c3ff64fc71d0ba not found: ID does not exist" Jul 06 19:17:16.217296 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.217260 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:17:16.220543 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:16.220520 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56bc5db5bb-ttfgp"] Jul 06 19:17:17.016675 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:17.016644 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" path="/var/lib/kubelet/pods/a328252d-e23e-4e49-9e96-7f753a7d9bd3/volumes" Jul 06 19:17:24.221136 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.221046 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d45d6d5c7-c4cdp" podUID="220568ce-7333-4f1d-80ad-bd2a625fd049" containerName="console" containerID="cri-o://6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039" gracePeriod=15 Jul 06 19:17:24.487384 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.487304 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:17:24.566437 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566403 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.566437 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566443 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.566667 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566482 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.566667 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566531 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.566667 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjtrx\" (UniqueName: \"kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.567086 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566865 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config" (OuterVolumeSpecName: "console-config") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:24.567086 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566879 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert\") pod \"220568ce-7333-4f1d-80ad-bd2a625fd049\" (UID: \"220568ce-7333-4f1d-80ad-bd2a625fd049\") " Jul 06 19:17:24.567086 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.566985 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca" (OuterVolumeSpecName: "service-ca") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:24.567303 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.567280 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-service-ca\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.567303 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.567300 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-console-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.567506 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.567489 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:17:24.568731 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.568707 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:24.568880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.568855 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx" (OuterVolumeSpecName: "kube-api-access-qjtrx") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "kube-api-access-qjtrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:24.568945 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.568906 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "220568ce-7333-4f1d-80ad-bd2a625fd049" (UID: "220568ce-7333-4f1d-80ad-bd2a625fd049"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:17:24.668088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.668053 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjtrx\" (UniqueName: \"kubernetes.io/projected/220568ce-7333-4f1d-80ad-bd2a625fd049-kube-api-access-qjtrx\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.668088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.668082 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/220568ce-7333-4f1d-80ad-bd2a625fd049-oauth-serving-cert\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.668088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.668095 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-serving-cert\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.668326 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:24.668103 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/220568ce-7333-4f1d-80ad-bd2a625fd049-console-oauth-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:17:25.235443 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.235408 2558 generic.go:356] "Generic (PLEG): container finished" podID="220568ce-7333-4f1d-80ad-bd2a625fd049" containerID="6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039" exitCode=0 Jul 06 19:17:25.235866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.235450 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d45d6d5c7-c4cdp" Jul 06 19:17:25.235866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.235476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d6d5c7-c4cdp" event={"ID":"220568ce-7333-4f1d-80ad-bd2a625fd049","Type":"ContainerDied","Data":"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039"} Jul 06 19:17:25.235866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.235517 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d45d6d5c7-c4cdp" event={"ID":"220568ce-7333-4f1d-80ad-bd2a625fd049","Type":"ContainerDied","Data":"b2171b9650723df2d108cb869ce389bacddc1ba95e231dfa2e71e254407db23c"} Jul 06 19:17:25.235866 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.235536 2558 scope.go:117] "RemoveContainer" containerID="6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039" Jul 06 19:17:25.243980 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.243963 2558 scope.go:117] "RemoveContainer" containerID="6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039" Jul 06 19:17:25.244223 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:17:25.244206 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039\": container with ID starting with 6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039 not found: ID does not exist" containerID="6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039" Jul 06 19:17:25.244279 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.244231 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039"} err="failed to get container status \"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039\": rpc error: code = NotFound desc = could not find container \"6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039\": container with ID starting with 6db4ba6c1ea15850ef50becf8b1b938fd8adea959fe32801e27a0a93e923b039 not found: ID does not exist" Jul 06 19:17:25.251862 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.251841 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:17:25.255541 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:25.255521 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d45d6d5c7-c4cdp"] Jul 06 19:17:27.017188 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:27.017146 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220568ce-7333-4f1d-80ad-bd2a625fd049" path="/var/lib/kubelet/pods/220568ce-7333-4f1d-80ad-bd2a625fd049/volumes" Jul 06 19:17:27.129329 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:27.129302 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:17:27.133255 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:27.133234 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-546989849d-wvpt5" Jul 06 19:17:33.272208 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:33.272177 2558 generic.go:356] "Generic (PLEG): container finished" podID="a54efb5e-c0ac-4ab6-99c4-0d833ab26d30" containerID="be9e4c8bbc020e93a08a8c93ff806462530d8d657272cf037255cc77ec6c7d3f" exitCode=0 Jul 06 19:17:33.272664 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:33.272220 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" event={"ID":"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30","Type":"ContainerDied","Data":"be9e4c8bbc020e93a08a8c93ff806462530d8d657272cf037255cc77ec6c7d3f"} Jul 06 19:17:33.272664 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:33.272557 2558 scope.go:117] "RemoveContainer" containerID="be9e4c8bbc020e93a08a8c93ff806462530d8d657272cf037255cc77ec6c7d3f" Jul 06 19:17:34.277729 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:34.277696 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-98lq4" event={"ID":"a54efb5e-c0ac-4ab6-99c4-0d833ab26d30","Type":"ContainerStarted","Data":"3df551bc7a7f5f9604cfc1b9dcb88c8e595fd0656e2cd2ad5db1d39e7eb8359f"} Jul 06 19:17:42.927781 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:42.927734 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:17:42.930257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:42.930231 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c193a56a-2f2c-4eac-86bd-e053d5b7bd57-metrics-certs\") pod \"network-metrics-daemon-q7p7g\" (UID: \"c193a56a-2f2c-4eac-86bd-e053d5b7bd57\") " pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:17:43.115115 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:43.115077 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r97mf\"" Jul 06 19:17:43.123037 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:43.123006 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7p7g" Jul 06 19:17:43.281877 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:43.281825 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7p7g"] Jul 06 19:17:43.284557 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:17:43.284529 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc193a56a_2f2c_4eac_86bd_e053d5b7bd57.slice/crio-d937aa46367f214bd799115abdf452d50fd08454060cade27678cbaebead1eab WatchSource:0}: Error finding container d937aa46367f214bd799115abdf452d50fd08454060cade27678cbaebead1eab: Status 404 returned error can't find the container with id d937aa46367f214bd799115abdf452d50fd08454060cade27678cbaebead1eab Jul 06 19:17:43.313839 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:43.313803 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7p7g" event={"ID":"c193a56a-2f2c-4eac-86bd-e053d5b7bd57","Type":"ContainerStarted","Data":"d937aa46367f214bd799115abdf452d50fd08454060cade27678cbaebead1eab"} Jul 06 19:17:45.324802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:45.324770 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7p7g" event={"ID":"c193a56a-2f2c-4eac-86bd-e053d5b7bd57","Type":"ContainerStarted","Data":"6da84617cdbc895247f9d2d0b4d66171a4ede0c39479540502f64f485f1ff2b7"} Jul 06 19:17:46.332900 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:46.332866 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7p7g" event={"ID":"c193a56a-2f2c-4eac-86bd-e053d5b7bd57","Type":"ContainerStarted","Data":"4e195e7591e6332a4c6be5f28fb952e12dc9e2a8a9fe9e172e3e317af7918341"} Jul 06 19:17:46.352270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:46.352221 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q7p7g" podStartSLOduration=375.488475634 podStartE2EDuration="6m17.352205139s" totalImagesPullingTime="1.863729505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:46.350991108 +0000 UTC m=+377.880336160" watchObservedRunningTime="2026-07-06 19:17:46.352205139 +0000 UTC m=+377.881550163" Jul 06 19:17:49.102836 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:49.102806 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:17:49.122216 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:49.122191 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:17:49.360523 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:17:49.360449 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.116875 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.116840 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117137 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="prometheus" containerID="cri-o://edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" gracePeriod=600 Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117173 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="config-reloader" containerID="cri-o://4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" gracePeriod=600 Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117201 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-web" containerID="cri-o://0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" gracePeriod=600 Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117242 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-thanos" containerID="cri-o://82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" gracePeriod=600 Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117173 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="thanos-sidecar" containerID="cri-o://daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" gracePeriod=600 Jul 06 19:18:07.117335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.117173 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy" containerID="cri-o://a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" gracePeriod=600 Jul 06 19:18:07.374498 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.374434 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.418221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418184 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" exitCode=0 Jul 06 19:18:07.418221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418215 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" exitCode=0 Jul 06 19:18:07.418221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418224 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" exitCode=0 Jul 06 19:18:07.418221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418233 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" exitCode=0 Jul 06 19:18:07.418221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418240 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" exitCode=0 Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418249 2558 generic.go:356] "Generic (PLEG): container finished" podID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" exitCode=0 Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418287 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418366 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418474 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418519 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418562 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418603 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} Jul 06 19:18:07.418641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418642 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} Jul 06 19:18:07.418983 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e986178d-1ba0-4d95-b652-4d911b37cf11","Type":"ContainerDied","Data":"504b8033c62a39f9a67679657a4dbf52efdb2af2dbbe6b9d3ae13679b4350520"} Jul 06 19:18:07.418983 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.418725 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.430753 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.430724 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.434569 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434543 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434582 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434613 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434640 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434664 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434688 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434716 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434739 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfp4x\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434775 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.434880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.434820 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436097 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436072 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436117 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436161 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436375 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436274 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436375 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436308 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436375 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436339 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436536 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436388 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436536 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436440 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.436536 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.436475 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls\") pod \"e986178d-1ba0-4d95-b652-4d911b37cf11\" (UID: \"e986178d-1ba0-4d95-b652-4d911b37cf11\") " Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.435159 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.435798 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.439339 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.439747 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.440132 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.440197 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.440884 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.441637 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.444299 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out" (OuterVolumeSpecName: "config-out") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:18:07.446211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.445560 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.446932 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.446901 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.447283 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447226 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:07.447397 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447286 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.447467 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447427 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.447587 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447550 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:07.447692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447580 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x" (OuterVolumeSpecName: "kube-api-access-nfp4x") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "kube-api-access-nfp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:07.448008 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447873 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.448008 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.447897 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.448257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.448229 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.449623 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.449597 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config" (OuterVolumeSpecName: "config") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.455806 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.455777 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.456281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.456262 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config" (OuterVolumeSpecName: "web-config") pod "e986178d-1ba0-4d95-b652-4d911b37cf11" (UID: "e986178d-1ba0-4d95-b652-4d911b37cf11"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:07.465108 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.465087 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.472457 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.472434 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.480157 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.480137 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.487034 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487018 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.487308 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.487289 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.487392 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487316 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.487392 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487336 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.487591 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.487574 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.487653 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487599 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.487653 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487621 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.487839 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.487825 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.487880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487842 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.487880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.487855 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.488065 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.488050 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.488106 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488069 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.488106 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488083 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.488276 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.488262 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.488378 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488279 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.488378 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488292 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.488498 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.488481 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.488538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488501 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.488538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488518 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.488740 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:07.488723 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.488784 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488744 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.488784 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488758 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.488978 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488948 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.489045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.488980 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.489235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489217 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.489296 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489235 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.489496 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489479 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.489543 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489498 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.489702 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489680 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.489775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489704 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.489949 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489932 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.490020 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.489950 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.490196 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490179 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.490269 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490197 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.490442 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490426 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.490490 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490446 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.490631 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490616 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.490667 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490632 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.490824 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490806 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.490869 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490825 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.491014 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.490998 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.491058 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491016 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.491241 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491223 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.491279 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491243 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.491526 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491504 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.491578 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491535 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.491778 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491760 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.491838 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491779 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.492007 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.491991 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.492051 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492009 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.492228 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492212 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.492266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492228 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.492472 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492453 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.492562 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492473 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.492707 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492689 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.492772 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492708 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.492947 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492931 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.493006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.492948 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.493159 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493144 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.493223 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493163 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.493402 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493380 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.493492 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493403 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.493608 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493591 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.493652 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493608 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.493821 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493802 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.493890 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.493823 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.494088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494070 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.494088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494086 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.494326 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494307 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.494427 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494326 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.494565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494550 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.494618 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494566 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.494775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494759 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.494862 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.494775 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.495021 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495002 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.495079 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495021 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.495270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495246 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.495270 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495270 2558 scope.go:117] "RemoveContainer" containerID="82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643" Jul 06 19:18:07.495506 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495490 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643"} err="failed to get container status \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": rpc error: code = NotFound desc = could not find container \"82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643\": container with ID starting with 82fac0f828b951e6ce69df810296266f39a601e1176acb582ac23bdd4c4bb643 not found: ID does not exist" Jul 06 19:18:07.495563 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495507 2558 scope.go:117] "RemoveContainer" containerID="a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9" Jul 06 19:18:07.495740 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495724 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9"} err="failed to get container status \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": rpc error: code = NotFound desc = could not find container \"a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9\": container with ID starting with a31f2aedf551302ac86f9e456761c9473b30f0599cbd45be7105b6c2e5c429c9 not found: ID does not exist" Jul 06 19:18:07.495789 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495740 2558 scope.go:117] "RemoveContainer" containerID="0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832" Jul 06 19:18:07.495963 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495935 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832"} err="failed to get container status \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": rpc error: code = NotFound desc = could not find container \"0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832\": container with ID starting with 0546deb1267e99e2a58af086ad9daac6ddeef74187dc915b3c4fe0756bc76832 not found: ID does not exist" Jul 06 19:18:07.495963 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.495953 2558 scope.go:117] "RemoveContainer" containerID="daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004" Jul 06 19:18:07.496182 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496155 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004"} err="failed to get container status \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": rpc error: code = NotFound desc = could not find container \"daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004\": container with ID starting with daffafe82601277b2718567df3848584cf54b50ee508badd5ee5b30e52db0004 not found: ID does not exist" Jul 06 19:18:07.496182 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496180 2558 scope.go:117] "RemoveContainer" containerID="4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac" Jul 06 19:18:07.496421 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496404 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac"} err="failed to get container status \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": rpc error: code = NotFound desc = could not find container \"4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac\": container with ID starting with 4d45648cce486eba0f73fb829d2ab0e5a56a2543908e6bcc5c0dab0420cbadac not found: ID does not exist" Jul 06 19:18:07.496421 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496421 2558 scope.go:117] "RemoveContainer" containerID="edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c" Jul 06 19:18:07.496631 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496612 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c"} err="failed to get container status \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": rpc error: code = NotFound desc = could not find container \"edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c\": container with ID starting with edea78395ac229587f836ee7f04264809726a70509ebc8fa85fa4679eb881d1c not found: ID does not exist" Jul 06 19:18:07.496631 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496631 2558 scope.go:117] "RemoveContainer" containerID="2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4" Jul 06 19:18:07.496816 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.496798 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4"} err="failed to get container status \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": rpc error: code = NotFound desc = could not find container \"2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4\": container with ID starting with 2767f4863b5303af4a640c917c720d3ab37d1f624cf106b4c401f5fcd0db57f4 not found: ID does not exist" Jul 06 19:18:07.537533 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537500 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537533 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537526 2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537533 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537538 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537547 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537557 2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-grpc-tls\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537566 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-config-out\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537574 2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-metrics-client-certs\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537582 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537591 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537598 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-tls-assets\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537606 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537614 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-web-config\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537622 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfp4x\" (UniqueName: \"kubernetes.io/projected/e986178d-1ba0-4d95-b652-4d911b37cf11-kube-api-access-nfp4x\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537630 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537638 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537647 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e986178d-1ba0-4d95-b652-4d911b37cf11-prometheus-k8s-db\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537655 2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-kube-rbac-proxy\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537662 2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537671 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e986178d-1ba0-4d95-b652-4d911b37cf11-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.537771 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.537679 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e986178d-1ba0-4d95-b652-4d911b37cf11-configmap-metrics-client-ca\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:18:07.749624 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.749587 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:07.757066 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.757040 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:07.785635 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785605 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:07.785903 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785890 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-web" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785903 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-web" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785914 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="init-config-reloader" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785919 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="init-config-reloader" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785930 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="thanos-sidecar" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785936 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="thanos-sidecar" Jul 06 19:18:07.785946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785945 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="config-reloader" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785951 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="config-reloader" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785957 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785961 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785969 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785974 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785981 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerName="registry" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785986 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerName="registry" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785991 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="prometheus" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.785995 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="prometheus" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786002 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-thanos" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786006 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-thanos" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786012 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220568ce-7333-4f1d-80ad-bd2a625fd049" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786018 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="220568ce-7333-4f1d-80ad-bd2a625fd049" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786071 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a328252d-e23e-4e49-9e96-7f753a7d9bd3" containerName="registry" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786078 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="prometheus" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786084 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-thanos" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786090 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6eccb27-4d22-4b06-9b8f-fac9593c979f" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786095 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="220568ce-7333-4f1d-80ad-bd2a625fd049" containerName="console" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786101 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="config-reloader" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786106 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786111 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="thanos-sidecar" Jul 06 19:18:07.786130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.786116 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" containerName="kube-rbac-proxy-web" Jul 06 19:18:07.791547 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.791527 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.794428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.794408 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:18:07.794544 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.794532 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:18:07.794609 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.794545 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtf9d\"" Jul 06 19:18:07.794818 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.794802 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:18:07.795528 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.795511 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:18:07.795716 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.795534 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:18:07.795716 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.795621 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:18:07.795883 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.795860 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:18:07.796134 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796115 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:18:07.796246 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796141 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:18:07.796246 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796143 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:18:07.796246 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796166 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ak0os96me7ume\"" Jul 06 19:18:07.796246 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796188 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:18:07.796538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.796520 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:18:07.799323 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.799301 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:18:07.802134 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.802118 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:18:07.812578 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.812535 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:07.838933 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.838904 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.838938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-config-out\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.838966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.838998 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839036 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839058 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839113 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839138 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839181 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839211 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839239 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-web-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839338 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839458 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839501 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xsr\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-kube-api-access-d4xsr\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839526 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.839654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.839569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940623 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940623 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940638 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-config-out\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940653 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940739 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940761 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940797 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940819 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.940865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940858 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940915 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-web-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940939 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940965 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.940987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941027 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941056 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941087 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xsr\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-kube-api-access-d4xsr\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.941248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941129 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942381 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941418 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942381 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941651 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942381 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.941782 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942381 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.942323 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942618 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.942441 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.942856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.942705 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.943441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.943415 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.944004 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.943982 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.944754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.944706 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-web-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.945113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.944997 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-config\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.945113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.945074 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5697d533-601e-40ba-881b-117dbc9ed917-config-out\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.945256 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.945217 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.945468 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.945443 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946280 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946261 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946520 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946494 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5697d533-601e-40ba-881b-117dbc9ed917-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946616 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946580 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946841 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946817 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946921 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946861 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.946962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.946931 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5697d533-601e-40ba-881b-117dbc9ed917-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:07.949713 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:07.949693 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xsr\" (UniqueName: \"kubernetes.io/projected/5697d533-601e-40ba-881b-117dbc9ed917-kube-api-access-d4xsr\") pod \"prometheus-k8s-0\" (UID: \"5697d533-601e-40ba-881b-117dbc9ed917\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:08.101302 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:08.101212 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:08.250537 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:08.250506 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:18:08.253817 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:18:08.253795 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5697d533_601e_40ba_881b_117dbc9ed917.slice/crio-4456add0a6a46e84707b49b3bd48aeaa7ca710f4fac9a551fbe8a1350898014c WatchSource:0}: Error finding container 4456add0a6a46e84707b49b3bd48aeaa7ca710f4fac9a551fbe8a1350898014c: Status 404 returned error can't find the container with id 4456add0a6a46e84707b49b3bd48aeaa7ca710f4fac9a551fbe8a1350898014c Jul 06 19:18:08.424465 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:08.424377 2558 generic.go:356] "Generic (PLEG): container finished" podID="5697d533-601e-40ba-881b-117dbc9ed917" containerID="afe65e75b1d662b0b60e179c0c9d087c640e9af0fa2251ec9c54c3fe40adf289" exitCode=0 Jul 06 19:18:08.424465 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:08.424422 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerDied","Data":"afe65e75b1d662b0b60e179c0c9d087c640e9af0fa2251ec9c54c3fe40adf289"} Jul 06 19:18:08.424654 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:08.424478 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"4456add0a6a46e84707b49b3bd48aeaa7ca710f4fac9a551fbe8a1350898014c"} Jul 06 19:18:09.018198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.018164 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e986178d-1ba0-4d95-b652-4d911b37cf11" path="/var/lib/kubelet/pods/e986178d-1ba0-4d95-b652-4d911b37cf11/volumes" Jul 06 19:18:09.435411 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435376 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"a043c412451b1ca184e8e13efc339e664f3e760f238d26c3b0e54c402c5a5a7e"} Jul 06 19:18:09.435815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435441 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"843c8a6b149c94fa972a574669a372547b5482dbf40b14f44d8653180469fb4f"} Jul 06 19:18:09.435815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"7560cc0968e3b4c4fc2d8b6afce75385dae76db35802b0377942d15f401ba376"} Jul 06 19:18:09.435815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435494 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"3a6a9688e75c989d95d75a95ee13a27195c3c20d90ce3c26903644c3a5b70215"} Jul 06 19:18:09.435815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435517 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"ce3e0beb79a0adfd08d0aeb05b7417a3a41abaf2df134c9d8f553784b47ba2d7"} Jul 06 19:18:09.435815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.435539 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5697d533-601e-40ba-881b-117dbc9ed917","Type":"ContainerStarted","Data":"297f3b6205d0158e4f64a0692586c0178a526b5ea440a29ad3d806cafac29bab"} Jul 06 19:18:09.467100 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:09.467039 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.444681142 podStartE2EDuration="2.467018087s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.336945ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:09.465994003 +0000 UTC m=+400.995339049" watchObservedRunningTime="2026-07-06 19:18:09.467018087 +0000 UTC m=+400.996363114" Jul 06 19:18:13.101997 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:13.101948 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:18:14.820902 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:18:14.820857 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" podUID="7be2f178-730b-4a7e-8a12-a163c0649821" Jul 06 19:18:15.459964 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:15.459935 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:18:17.330195 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:17.330146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:18:17.332659 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:17.332632 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7be2f178-730b-4a7e-8a12-a163c0649821-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-4rglq\" (UID: \"7be2f178-730b-4a7e-8a12-a163c0649821\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:18:17.562833 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:17.562804 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g22xl\"" Jul 06 19:18:17.571282 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:17.571258 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" Jul 06 19:18:17.741519 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:17.741496 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-4rglq"] Jul 06 19:18:17.744176 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:18:17.744145 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be2f178_730b_4a7e_8a12_a163c0649821.slice/crio-3497c5397efdd7ccdb57581fbe9d29a07c3714e1a8390d9ace68941db1e308e4 WatchSource:0}: Error finding container 3497c5397efdd7ccdb57581fbe9d29a07c3714e1a8390d9ace68941db1e308e4: Status 404 returned error can't find the container with id 3497c5397efdd7ccdb57581fbe9d29a07c3714e1a8390d9ace68941db1e308e4 Jul 06 19:18:18.473896 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:18.473863 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" event={"ID":"7be2f178-730b-4a7e-8a12-a163c0649821","Type":"ContainerStarted","Data":"3497c5397efdd7ccdb57581fbe9d29a07c3714e1a8390d9ace68941db1e308e4"} Jul 06 19:18:19.479290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:19.479256 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" event={"ID":"7be2f178-730b-4a7e-8a12-a163c0649821","Type":"ContainerStarted","Data":"93dbcc99564b5514a7ca05f430d9974216147aba7a1689c3bb2e8ab73a82b75e"} Jul 06 19:18:19.497386 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:18:19.497319 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-4rglq" podStartSLOduration=382.188114514 podStartE2EDuration="6m23.497304616s" totalImagesPullingTime="1.309190102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:19.496036019 +0000 UTC m=+411.025381043" watchObservedRunningTime="2026-07-06 19:18:19.497304616 +0000 UTC m=+411.026649647" Jul 06 19:19:08.102367 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:19:08.102313 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:19:08.118303 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:19:08.118276 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:19:08.706823 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:19:08.706795 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:21:08.781257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.781223 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qqz2k"] Jul 06 19:21:08.784500 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.784483 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:08.787088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.787065 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-csmlc\"" Jul 06 19:21:08.787904 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.787889 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:21:08.787975 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.787892 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:21:08.793816 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.793796 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qqz2k"] Jul 06 19:21:08.956833 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.956797 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f62\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-kube-api-access-l5f62\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:08.957005 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:08.956841 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.058123 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.058050 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f62\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-kube-api-access-l5f62\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.058123 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.058088 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.082624 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.082592 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.082959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.082941 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f62\" (UniqueName: \"kubernetes.io/projected/1b44d53f-1c66-460e-9950-3b0fa74e200f-kube-api-access-l5f62\") pod \"cert-manager-webhook-6486ffd997-qqz2k\" (UID: \"1b44d53f-1c66-460e-9950-3b0fa74e200f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.101233 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.101211 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:09.258683 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:09.258657 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qqz2k"] Jul 06 19:21:09.261339 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:21:09.261310 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b44d53f_1c66_460e_9950_3b0fa74e200f.slice/crio-b1342a1b09d8166e91d9b26d0d2c007d8a04d0b0a31f25e887d3c35ab366e171 WatchSource:0}: Error finding container b1342a1b09d8166e91d9b26d0d2c007d8a04d0b0a31f25e887d3c35ab366e171: Status 404 returned error can't find the container with id b1342a1b09d8166e91d9b26d0d2c007d8a04d0b0a31f25e887d3c35ab366e171 Jul 06 19:21:10.174902 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:10.174861 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" event={"ID":"1b44d53f-1c66-460e-9950-3b0fa74e200f","Type":"ContainerStarted","Data":"b1342a1b09d8166e91d9b26d0d2c007d8a04d0b0a31f25e887d3c35ab366e171"} Jul 06 19:21:13.190622 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:13.190583 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" event={"ID":"1b44d53f-1c66-460e-9950-3b0fa74e200f","Type":"ContainerStarted","Data":"69f95c160d9e09f6b75bc42bc3a6bc3c7d453b6a8bd785711805abb79f7b9e8a"} Jul 06 19:21:13.191013 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:13.190711 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:13.206724 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:13.206680 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" podStartSLOduration=2.261820193 podStartE2EDuration="5.206666581s" totalImagesPullingTime="2.944846388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:13.206438389 +0000 UTC m=+584.735783414" watchObservedRunningTime="2026-07-06 19:21:13.206666581 +0000 UTC m=+584.736011607" Jul 06 19:21:19.196950 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:19.196921 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qqz2k" Jul 06 19:21:28.921796 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:28.921761 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:21:28.922210 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:21:28.922047 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:22:15.150494 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.150460 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-gw8n8"] Jul 06 19:22:15.153786 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.153770 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.156765 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.156745 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4pdhd\"" Jul 06 19:22:15.157656 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.157641 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:22:15.158056 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.158035 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:22:15.158056 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.158056 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:22:15.158197 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.158037 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:22:15.158197 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.158074 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:22:15.168337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.168315 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-gw8n8"] Jul 06 19:22:15.213324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.213299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-metrics-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.213456 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.213328 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thqt\" (UniqueName: \"kubernetes.io/projected/d036056a-6067-4163-ab3c-af4a384496ff-kube-api-access-8thqt\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.213456 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.213377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d036056a-6067-4163-ab3c-af4a384496ff-manager-config\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.213456 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.213401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.314020 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.313983 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-metrics-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.314020 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.314021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thqt\" (UniqueName: \"kubernetes.io/projected/d036056a-6067-4163-ab3c-af4a384496ff-kube-api-access-8thqt\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.314201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.314049 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d036056a-6067-4163-ab3c-af4a384496ff-manager-config\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.314201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.314164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.314803 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.314765 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d036056a-6067-4163-ab3c-af4a384496ff-manager-config\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.316844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.316821 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-metrics-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.316844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.316839 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d036056a-6067-4163-ab3c-af4a384496ff-cert\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.326094 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.326071 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thqt\" (UniqueName: \"kubernetes.io/projected/d036056a-6067-4163-ab3c-af4a384496ff-kube-api-access-8thqt\") pod \"lws-controller-manager-665d858957-gw8n8\" (UID: \"d036056a-6067-4163-ab3c-af4a384496ff\") " pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.462766 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.462738 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:15.594097 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.594065 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-665d858957-gw8n8"] Jul 06 19:22:15.596982 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:22:15.596956 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd036056a_6067_4163_ab3c_af4a384496ff.slice/crio-0f2435bb89d016c340c8068dd0e5e02d18bc6a1668dfd43bc95c59f5248dc13f WatchSource:0}: Error finding container 0f2435bb89d016c340c8068dd0e5e02d18bc6a1668dfd43bc95c59f5248dc13f: Status 404 returned error can't find the container with id 0f2435bb89d016c340c8068dd0e5e02d18bc6a1668dfd43bc95c59f5248dc13f Jul 06 19:22:15.598800 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:15.598783 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:22:16.456118 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:16.456075 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" event={"ID":"d036056a-6067-4163-ab3c-af4a384496ff","Type":"ContainerStarted","Data":"0f2435bb89d016c340c8068dd0e5e02d18bc6a1668dfd43bc95c59f5248dc13f"} Jul 06 19:22:18.466791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:18.466755 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" event={"ID":"d036056a-6067-4163-ab3c-af4a384496ff","Type":"ContainerStarted","Data":"58bc01845809727b1cd4d17b1412d81eddcabd3e42118c2c9f03bf6540a4251c"} Jul 06 19:22:18.467153 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:18.466857 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:18.485294 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:18.485247 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" podStartSLOduration=1.054104598 podStartE2EDuration="3.485231444s" totalImagesPullingTime="2.431126846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:18.483929228 +0000 UTC m=+650.013274266" watchObservedRunningTime="2026-07-06 19:22:18.485231444 +0000 UTC m=+650.014576468" Jul 06 19:22:26.291971 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.291934 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9"] Jul 06 19:22:26.295342 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.295325 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.300962 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.300940 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:22:26.301793 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.301527 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jdd5t\"" Jul 06 19:22:26.301793 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.301662 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:22:26.301793 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.301686 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:22:26.301793 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.301782 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:22:26.319405 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.319372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9"] Jul 06 19:22:26.406902 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.406864 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.406902 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.406903 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.407116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.406927 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.407116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.407011 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.407116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.407059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqdl\" (UniqueName: \"kubernetes.io/projected/b127174a-ba46-42a3-b0ca-190ed348baae-kube-api-access-fdqdl\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.507876 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.507845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.507876 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.507877 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.508093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.507916 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.508093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.507947 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.508093 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.507967 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqdl\" (UniqueName: \"kubernetes.io/projected/b127174a-ba46-42a3-b0ca-190ed348baae-kube-api-access-fdqdl\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.508314 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.508290 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-manifests\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.510568 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.510543 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b127174a-ba46-42a3-b0ca-190ed348baae-tmp\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.510635 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.510595 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.510672 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.510640 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b127174a-ba46-42a3-b0ca-190ed348baae-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.519464 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.519435 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqdl\" (UniqueName: \"kubernetes.io/projected/b127174a-ba46-42a3-b0ca-190ed348baae-kube-api-access-fdqdl\") pod \"opendatahub-operator-controller-manager-56f978bf57-j4bt9\" (UID: \"b127174a-ba46-42a3-b0ca-190ed348baae\") " pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.606193 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.606103 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:26.742069 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:26.742034 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9"] Jul 06 19:22:26.745312 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:22:26.745288 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb127174a_ba46_42a3_b0ca_190ed348baae.slice/crio-3c62772ebbeeeaef1d10898426619d628d0ee146dd8177e628b64a5c6d105b1f WatchSource:0}: Error finding container 3c62772ebbeeeaef1d10898426619d628d0ee146dd8177e628b64a5c6d105b1f: Status 404 returned error can't find the container with id 3c62772ebbeeeaef1d10898426619d628d0ee146dd8177e628b64a5c6d105b1f Jul 06 19:22:27.512005 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:27.511895 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" event={"ID":"b127174a-ba46-42a3-b0ca-190ed348baae","Type":"ContainerStarted","Data":"3c62772ebbeeeaef1d10898426619d628d0ee146dd8177e628b64a5c6d105b1f"} Jul 06 19:22:29.473141 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:29.473108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-665d858957-gw8n8" Jul 06 19:22:30.527298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:30.527260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" event={"ID":"b127174a-ba46-42a3-b0ca-190ed348baae","Type":"ContainerStarted","Data":"3baeeec968eb63a7b7ed9b1089cd61fac32a6e04d1c7b568a57e5501e5781e72"} Jul 06 19:22:31.533323 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:31.533233 2558 generic.go:356] "Generic (PLEG): container finished" podID="b127174a-ba46-42a3-b0ca-190ed348baae" containerID="3baeeec968eb63a7b7ed9b1089cd61fac32a6e04d1c7b568a57e5501e5781e72" exitCode=0 Jul 06 19:22:31.533323 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:31.533302 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" event={"ID":"b127174a-ba46-42a3-b0ca-190ed348baae","Type":"ContainerDied","Data":"3baeeec968eb63a7b7ed9b1089cd61fac32a6e04d1c7b568a57e5501e5781e72"} Jul 06 19:22:32.541844 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:32.541806 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" event={"ID":"b127174a-ba46-42a3-b0ca-190ed348baae","Type":"ContainerStarted","Data":"93b3a7e62cfd978c707fe3e3f61d3c4581648c4bcf0ce8c8c66b46012f706941"} Jul 06 19:22:32.542236 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:32.541926 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:32.562484 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:32.562435 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" podStartSLOduration=1.690314654 podStartE2EDuration="6.562419289s" totalImagesPullingTime="3.808602679s" totalInitContainerRuntime="1.063501956s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:32.561477877 +0000 UTC m=+664.090822917" watchObservedRunningTime="2026-07-06 19:22:32.562419289 +0000 UTC m=+664.091764314" Jul 06 19:22:43.549853 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:43.549818 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56f978bf57-j4bt9" Jul 06 19:22:47.603829 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.603796 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9"] Jul 06 19:22:47.620416 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.620376 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.622674 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.622647 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:22:47.623450 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.623430 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:22:47.624017 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.623993 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:22:47.624922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.624632 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:22:47.624922 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.624795 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cgbq9\"" Jul 06 19:22:47.630770 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.630747 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9"] Jul 06 19:22:47.706315 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.706279 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99602cc-8c62-49d9-bb00-11c6c336167c-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.706515 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.706364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c99602cc-8c62-49d9-bb00-11c6c336167c-tmp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.706515 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.706390 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpfp\" (UniqueName: \"kubernetes.io/projected/c99602cc-8c62-49d9-bb00-11c6c336167c-kube-api-access-qxpfp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.807699 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.807665 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99602cc-8c62-49d9-bb00-11c6c336167c-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.807885 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.807727 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c99602cc-8c62-49d9-bb00-11c6c336167c-tmp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.807885 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.807753 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpfp\" (UniqueName: \"kubernetes.io/projected/c99602cc-8c62-49d9-bb00-11c6c336167c-kube-api-access-qxpfp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.810340 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.810310 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c99602cc-8c62-49d9-bb00-11c6c336167c-tmp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.810557 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.810537 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99602cc-8c62-49d9-bb00-11c6c336167c-tls-certs\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.817070 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.817043 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpfp\" (UniqueName: \"kubernetes.io/projected/c99602cc-8c62-49d9-bb00-11c6c336167c-kube-api-access-qxpfp\") pod \"kube-auth-proxy-6fffdf4997-zbhw9\" (UID: \"c99602cc-8c62-49d9-bb00-11c6c336167c\") " pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:47.936259 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:47.936219 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" Jul 06 19:22:48.095385 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:48.095336 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9"] Jul 06 19:22:48.098443 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:22:48.098404 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99602cc_8c62_49d9_bb00_11c6c336167c.slice/crio-beeb25d4f4a81f47a919e9e2723c835c942a0ae2581ea51bfc470399b2436ce0 WatchSource:0}: Error finding container beeb25d4f4a81f47a919e9e2723c835c942a0ae2581ea51bfc470399b2436ce0: Status 404 returned error can't find the container with id beeb25d4f4a81f47a919e9e2723c835c942a0ae2581ea51bfc470399b2436ce0 Jul 06 19:22:48.623418 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:48.623373 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" event={"ID":"c99602cc-8c62-49d9-bb00-11c6c336167c","Type":"ContainerStarted","Data":"beeb25d4f4a81f47a919e9e2723c835c942a0ae2581ea51bfc470399b2436ce0"} Jul 06 19:22:51.640585 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:51.640495 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" event={"ID":"c99602cc-8c62-49d9-bb00-11c6c336167c","Type":"ContainerStarted","Data":"68f06a8c1daedff4303f217f1d90070304d231bae17a4dc2437c61fa1adaee81"} Jul 06 19:22:51.669696 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:22:51.669640 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fffdf4997-zbhw9" podStartSLOduration=1.415377083 podStartE2EDuration="4.66962498s" totalImagesPullingTime="3.254247897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:51.668785302 +0000 UTC m=+683.198130328" watchObservedRunningTime="2026-07-06 19:22:51.66962498 +0000 UTC m=+683.198970004" Jul 06 19:23:15.397784 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.397751 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786f59bffc-sxfkt"] Jul 06 19:23:15.400749 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.400732 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.403033 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.403005 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:23:15.403856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.403838 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8p8r\"" Jul 06 19:23:15.403948 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.403897 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:23:15.404000 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.403984 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:23:15.404054 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.404021 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:23:15.404240 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.404223 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:23:15.408897 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.408876 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:23:15.415252 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.415231 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786f59bffc-sxfkt"] Jul 06 19:23:15.441746 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441722 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-trusted-ca-bundle\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441854 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441751 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-oauth-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441854 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441769 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-service-ca\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441854 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-oauth-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441907 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcfr\" (UniqueName: \"kubernetes.io/projected/ca877ba2-25a7-4844-9fff-cce512f3a830-kube-api-access-rqcfr\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.441973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.441957 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-console-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.542775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542742 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.542775 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542781 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcfr\" (UniqueName: \"kubernetes.io/projected/ca877ba2-25a7-4844-9fff-cce512f3a830-kube-api-access-rqcfr\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543001 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542867 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-console-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543001 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-trusted-ca-bundle\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543001 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542934 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-oauth-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543001 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542955 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-service-ca\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543001 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.542977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-oauth-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543689 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.543665 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-oauth-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.543833 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-trusted-ca-bundle\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543911 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.543857 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-service-ca\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.543911 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.543869 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca877ba2-25a7-4844-9fff-cce512f3a830-console-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.546143 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.546120 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-oauth-config\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.546236 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.546126 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca877ba2-25a7-4844-9fff-cce512f3a830-console-serving-cert\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.550180 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.550161 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcfr\" (UniqueName: \"kubernetes.io/projected/ca877ba2-25a7-4844-9fff-cce512f3a830-kube-api-access-rqcfr\") pod \"console-786f59bffc-sxfkt\" (UID: \"ca877ba2-25a7-4844-9fff-cce512f3a830\") " pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.710864 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.710832 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:15.843409 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:15.843382 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786f59bffc-sxfkt"] Jul 06 19:23:15.845330 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:23:15.845297 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca877ba2_25a7_4844_9fff_cce512f3a830.slice/crio-2faa3134c0fb3db806cfc4160f06b9742ea5de39b80839d3a0c822464a7cc9e7 WatchSource:0}: Error finding container 2faa3134c0fb3db806cfc4160f06b9742ea5de39b80839d3a0c822464a7cc9e7: Status 404 returned error can't find the container with id 2faa3134c0fb3db806cfc4160f06b9742ea5de39b80839d3a0c822464a7cc9e7 Jul 06 19:23:16.757373 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:16.757317 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f59bffc-sxfkt" event={"ID":"ca877ba2-25a7-4844-9fff-cce512f3a830","Type":"ContainerStarted","Data":"85d2a1bf60f12646ab4eea5c94d2ad3881983feab0ef24585374259d96fd3fe4"} Jul 06 19:23:16.757760 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:16.757394 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786f59bffc-sxfkt" event={"ID":"ca877ba2-25a7-4844-9fff-cce512f3a830","Type":"ContainerStarted","Data":"2faa3134c0fb3db806cfc4160f06b9742ea5de39b80839d3a0c822464a7cc9e7"} Jul 06 19:23:16.775246 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:16.775195 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786f59bffc-sxfkt" podStartSLOduration=1.775179445 podStartE2EDuration="1.775179445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:23:16.773540334 +0000 UTC m=+708.302885383" watchObservedRunningTime="2026-07-06 19:23:16.775179445 +0000 UTC m=+708.304524470" Jul 06 19:23:25.711916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:25.711821 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:25.711916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:25.711898 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:25.716278 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:25.716258 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:23:25.807565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:23:25.807534 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786f59bffc-sxfkt" Jul 06 19:24:33.105945 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.105915 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd"] Jul 06 19:24:33.109564 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.109542 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.112507 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.112488 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:24:33.112871 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.112848 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:24:33.112970 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.112874 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jfkvd\"" Jul 06 19:24:33.113307 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.113293 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:24:33.113387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.113336 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:24:33.117789 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.117771 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd"] Jul 06 19:24:33.135586 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.135560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba70de49-01a9-40ec-81af-9a0dc112bfc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.135692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.135605 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhtn\" (UniqueName: \"kubernetes.io/projected/ba70de49-01a9-40ec-81af-9a0dc112bfc2-kube-api-access-4jhtn\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.135692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.135631 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.236961 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.236932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhtn\" (UniqueName: \"kubernetes.io/projected/ba70de49-01a9-40ec-81af-9a0dc112bfc2-kube-api-access-4jhtn\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.237109 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.236977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.237109 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.237061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba70de49-01a9-40ec-81af-9a0dc112bfc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.237219 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:24:33.237142 2558 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 19:24:33.237278 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:24:33.237227 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert podName:ba70de49-01a9-40ec-81af-9a0dc112bfc2 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:33.737200854 +0000 UTC m=+785.266545862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-nxlcd" (UID: "ba70de49-01a9-40ec-81af-9a0dc112bfc2") : secret "plugin-serving-cert" not found Jul 06 19:24:33.237747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.237725 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ba70de49-01a9-40ec-81af-9a0dc112bfc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.245999 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.245972 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhtn\" (UniqueName: \"kubernetes.io/projected/ba70de49-01a9-40ec-81af-9a0dc112bfc2-kube-api-access-4jhtn\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.741732 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.741701 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:33.744130 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:33.744109 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ba70de49-01a9-40ec-81af-9a0dc112bfc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nxlcd\" (UID: \"ba70de49-01a9-40ec-81af-9a0dc112bfc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:34.019783 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:34.019692 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" Jul 06 19:24:34.159369 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:34.159328 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd"] Jul 06 19:24:34.161429 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:24:34.161398 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba70de49_01a9_40ec_81af_9a0dc112bfc2.slice/crio-102f62cb32bff169a9dcd5db39764240e7a700be53738e29f31a5386d2888939 WatchSource:0}: Error finding container 102f62cb32bff169a9dcd5db39764240e7a700be53738e29f31a5386d2888939: Status 404 returned error can't find the container with id 102f62cb32bff169a9dcd5db39764240e7a700be53738e29f31a5386d2888939 Jul 06 19:24:35.117797 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:35.117757 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" event={"ID":"ba70de49-01a9-40ec-81af-9a0dc112bfc2","Type":"ContainerStarted","Data":"102f62cb32bff169a9dcd5db39764240e7a700be53738e29f31a5386d2888939"} Jul 06 19:24:58.246503 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:58.246467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" event={"ID":"ba70de49-01a9-40ec-81af-9a0dc112bfc2","Type":"ContainerStarted","Data":"e1f007f0cdc2e1d8878635436be08d613ef8cc46081823de78716c45b5e51c7b"} Jul 06 19:24:58.271478 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:24:58.271429 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nxlcd" podStartSLOduration=1.770234468 podStartE2EDuration="25.271415626s" totalImagesPullingTime="23.501181158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:58.269085426 +0000 UTC m=+809.798430467" watchObservedRunningTime="2026-07-06 19:24:58.271415626 +0000 UTC m=+809.800760650" Jul 06 19:25:24.628066 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:24.628033 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:25:25.055588 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.055550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:25:25.055791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.055576 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.055791 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.055622 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:25:25.058152 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.058130 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:25:25.115533 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.115503 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7r6p\" (UniqueName: \"kubernetes.io/projected/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-kube-api-access-h7r6p\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.115701 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.115680 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-config-file\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.217011 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.216973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-config-file\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.217184 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.217061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7r6p\" (UniqueName: \"kubernetes.io/projected/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-kube-api-access-h7r6p\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.217632 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.217614 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-config-file\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.220954 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.220928 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:25:25.234313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.234287 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7r6p\" (UniqueName: \"kubernetes.io/projected/cf6b915b-45d1-4fe8-92d4-b3f190f924cd-kube-api-access-h7r6p\") pod \"limitador-limitador-54d6d988d5-fcxhd\" (UID: \"cf6b915b-45d1-4fe8-92d4-b3f190f924cd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.334994 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.334919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:25:25.335139 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.335008 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:25:25.337636 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.337612 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fvqqr\"" Jul 06 19:25:25.367322 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.367297 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:25.419602 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.419573 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvht\" (UniqueName: \"kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht\") pod \"authorino-75d556bd77-7dwhc\" (UID: \"293ed95d-0a55-4d3f-b67b-43289b152b91\") " pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:25:25.520467 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.520432 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvht\" (UniqueName: \"kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht\") pod \"authorino-75d556bd77-7dwhc\" (UID: \"293ed95d-0a55-4d3f-b67b-43289b152b91\") " pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:25:25.526638 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.526611 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:25:25.529794 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:25:25.529756 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6b915b_45d1_4fe8_92d4_b3f190f924cd.slice/crio-d4693a96a46751e98741cd97007d928f9c958c639f1a7374c96d32b23933c2e8 WatchSource:0}: Error finding container d4693a96a46751e98741cd97007d928f9c958c639f1a7374c96d32b23933c2e8: Status 404 returned error can't find the container with id d4693a96a46751e98741cd97007d928f9c958c639f1a7374c96d32b23933c2e8 Jul 06 19:25:25.530432 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.530405 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvht\" (UniqueName: \"kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht\") pod \"authorino-75d556bd77-7dwhc\" (UID: \"293ed95d-0a55-4d3f-b67b-43289b152b91\") " pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:25:25.645638 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.645558 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:25:25.785873 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:25.785819 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:25:25.788060 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:25:25.788034 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod293ed95d_0a55_4d3f_b67b_43289b152b91.slice/crio-97e9bcc269739f7c4963161e4c927cea56be4492c584b742ea0f443452e690e9 WatchSource:0}: Error finding container 97e9bcc269739f7c4963161e4c927cea56be4492c584b742ea0f443452e690e9: Status 404 returned error can't find the container with id 97e9bcc269739f7c4963161e4c927cea56be4492c584b742ea0f443452e690e9 Jul 06 19:25:26.374690 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:26.374651 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7dwhc" event={"ID":"293ed95d-0a55-4d3f-b67b-43289b152b91","Type":"ContainerStarted","Data":"97e9bcc269739f7c4963161e4c927cea56be4492c584b742ea0f443452e690e9"} Jul 06 19:25:26.376058 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:26.376030 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" event={"ID":"cf6b915b-45d1-4fe8-92d4-b3f190f924cd","Type":"ContainerStarted","Data":"d4693a96a46751e98741cd97007d928f9c958c639f1a7374c96d32b23933c2e8"} Jul 06 19:25:31.410198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:31.410099 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7dwhc" event={"ID":"293ed95d-0a55-4d3f-b67b-43289b152b91","Type":"ContainerStarted","Data":"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973"} Jul 06 19:25:31.411607 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:31.411584 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" event={"ID":"cf6b915b-45d1-4fe8-92d4-b3f190f924cd","Type":"ContainerStarted","Data":"5e186a6643cf839f0ec6c65867a33b3e4e3992b4218a29305c38c73a0af00346"} Jul 06 19:25:31.411738 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:31.411722 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:31.449728 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:31.449684 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-7dwhc" podStartSLOduration=1.256172466 podStartE2EDuration="6.449670028s" totalImagesPullingTime="5.193497562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:31.446672745 +0000 UTC m=+842.976017770" watchObservedRunningTime="2026-07-06 19:25:31.449670028 +0000 UTC m=+842.979015053" Jul 06 19:25:42.419488 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:42.419455 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" Jul 06 19:25:42.441045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:25:42.440998 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-fcxhd" podStartSLOduration=13.036971086 podStartE2EDuration="18.440984268s" totalImagesPullingTime="5.404013182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:31.514156365 +0000 UTC m=+843.043501393" watchObservedRunningTime="2026-07-06 19:25:42.440984268 +0000 UTC m=+853.970329342" Jul 06 19:26:03.250116 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.250077 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:26:03.250560 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.250284 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-7dwhc" podUID="293ed95d-0a55-4d3f-b67b-43289b152b91" containerName="authorino" containerID="cri-o://f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973" gracePeriod=30 Jul 06 19:26:03.503961 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.503900 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:26:03.558077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.558038 2558 generic.go:356] "Generic (PLEG): container finished" podID="293ed95d-0a55-4d3f-b67b-43289b152b91" containerID="f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973" exitCode=0 Jul 06 19:26:03.558077 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.558070 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7dwhc" Jul 06 19:26:03.558259 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.558110 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7dwhc" event={"ID":"293ed95d-0a55-4d3f-b67b-43289b152b91","Type":"ContainerDied","Data":"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973"} Jul 06 19:26:03.558259 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.558159 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7dwhc" event={"ID":"293ed95d-0a55-4d3f-b67b-43289b152b91","Type":"ContainerDied","Data":"97e9bcc269739f7c4963161e4c927cea56be4492c584b742ea0f443452e690e9"} Jul 06 19:26:03.558259 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.558177 2558 scope.go:117] "RemoveContainer" containerID="f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973" Jul 06 19:26:03.561627 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.561605 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsvht\" (UniqueName: \"kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht\") pod \"293ed95d-0a55-4d3f-b67b-43289b152b91\" (UID: \"293ed95d-0a55-4d3f-b67b-43289b152b91\") " Jul 06 19:26:03.564071 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.564039 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht" (OuterVolumeSpecName: "kube-api-access-gsvht") pod "293ed95d-0a55-4d3f-b67b-43289b152b91" (UID: "293ed95d-0a55-4d3f-b67b-43289b152b91"). InnerVolumeSpecName "kube-api-access-gsvht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:26:03.571207 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.571189 2558 scope.go:117] "RemoveContainer" containerID="f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973" Jul 06 19:26:03.572006 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:26:03.571922 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973\": container with ID starting with f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973 not found: ID does not exist" containerID="f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973" Jul 06 19:26:03.572006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.571973 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973"} err="failed to get container status \"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973\": rpc error: code = NotFound desc = could not find container \"f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973\": container with ID starting with f40035d196f77996b0fc6092213fda821a670b2c766c572a6c1f93d7eef68973 not found: ID does not exist" Jul 06 19:26:03.662987 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.662958 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvht\" (UniqueName: \"kubernetes.io/projected/293ed95d-0a55-4d3f-b67b-43289b152b91-kube-api-access-gsvht\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:26:03.887744 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.887711 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:26:03.900809 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:03.900789 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7dwhc"] Jul 06 19:26:05.017937 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:05.017904 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="293ed95d-0a55-4d3f-b67b-43289b152b91" path="/var/lib/kubelet/pods/293ed95d-0a55-4d3f-b67b-43289b152b91/volumes" Jul 06 19:26:28.957792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:28.957722 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:26:28.958862 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:28.958844 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:26:42.535917 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:26:42.535881 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:27:05.976857 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.976818 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk"] Jul 06 19:27:05.977314 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.977214 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="293ed95d-0a55-4d3f-b67b-43289b152b91" containerName="authorino" Jul 06 19:27:05.977314 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.977228 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="293ed95d-0a55-4d3f-b67b-43289b152b91" containerName="authorino" Jul 06 19:27:05.977314 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.977312 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="293ed95d-0a55-4d3f-b67b-43289b152b91" containerName="authorino" Jul 06 19:27:05.980428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.980412 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:05.983018 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.982997 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h6bns\"" Jul 06 19:27:05.983495 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.983478 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:27:05.984146 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.984130 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 19:27:05.984214 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.984181 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:27:05.993021 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:05.992996 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk"] Jul 06 19:27:06.096124 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096089 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wrf\" (UniqueName: \"kubernetes.io/projected/c3913cab-6dbd-44ed-99e5-1679ece01530-kube-api-access-g4wrf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.096290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096133 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.096290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096198 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.096290 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.096498 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3913cab-6dbd-44ed-99e5-1679ece01530-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.096498 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.096372 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.196864 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196831 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197030 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197030 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3913cab-6dbd-44ed-99e5-1679ece01530-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197030 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197030 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wrf\" (UniqueName: \"kubernetes.io/projected/c3913cab-6dbd-44ed-99e5-1679ece01530-kube-api-access-g4wrf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197030 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.196988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.197274 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.197319 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.197446 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.197405 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.199310 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.199293 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3913cab-6dbd-44ed-99e5-1679ece01530-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.199595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.199578 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3913cab-6dbd-44ed-99e5-1679ece01530-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.205928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.205907 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wrf\" (UniqueName: \"kubernetes.io/projected/c3913cab-6dbd-44ed-99e5-1679ece01530-kube-api-access-g4wrf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk\" (UID: \"c3913cab-6dbd-44ed-99e5-1679ece01530\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.290674 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.290568 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:06.425830 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.425792 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk"] Jul 06 19:27:06.429116 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:27:06.429091 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3913cab_6dbd_44ed_99e5_1679ece01530.slice/crio-5b062fa147a6e3b549cb5863729cb28e393d79f301b93901a9aa20387bf3235d WatchSource:0}: Error finding container 5b062fa147a6e3b549cb5863729cb28e393d79f301b93901a9aa20387bf3235d: Status 404 returned error can't find the container with id 5b062fa147a6e3b549cb5863729cb28e393d79f301b93901a9aa20387bf3235d Jul 06 19:27:06.429268 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.429121 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:27:06.863628 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:06.863495 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" event={"ID":"c3913cab-6dbd-44ed-99e5-1679ece01530","Type":"ContainerStarted","Data":"5b062fa147a6e3b549cb5863729cb28e393d79f301b93901a9aa20387bf3235d"} Jul 06 19:27:07.873335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:07.873292 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" event={"ID":"c3913cab-6dbd-44ed-99e5-1679ece01530","Type":"ContainerStarted","Data":"9705a3386f0fbeaafa2c13c40e1afa1fe9421c987287a20467c8c404fcf043a2"} Jul 06 19:27:07.894597 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:07.894546 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" podStartSLOduration=1.637804407 podStartE2EDuration="2.89453053s" totalImagesPullingTime="1.256726123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:07.891973873 +0000 UTC m=+939.421318908" watchObservedRunningTime="2026-07-06 19:27:07.89453053 +0000 UTC m=+939.423875555" Jul 06 19:27:12.409832 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:12.409788 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:27:16.291025 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:16.290982 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:16.291457 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:16.291053 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:16.303979 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:16.303951 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:16.924682 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:16.924653 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk" Jul 06 19:27:25.463613 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:25.463571 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:27:47.859380 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:47.859335 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:27:58.018700 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.018624 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824"] Jul 06 19:27:58.021979 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.021963 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.024755 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.024727 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:27:58.047977 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.047948 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824"] Jul 06 19:27:58.156746 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.156912 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156762 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.156912 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156791 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2hr\" (UniqueName: \"kubernetes.io/projected/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-kube-api-access-sg2hr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.156912 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156861 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.156912 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156908 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.157045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.156947 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258205 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258173 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258419 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258213 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2hr\" (UniqueName: \"kubernetes.io/projected/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-kube-api-access-sg2hr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258419 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258254 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258419 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258419 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258325 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258419 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258688 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258593 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258688 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258667 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.258800 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.258689 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.260635 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.260609 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.260899 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.260879 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.273824 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.273760 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2hr\" (UniqueName: \"kubernetes.io/projected/fa2eaa44-a12c-4056-9343-b0dbea9a57ff-kube-api-access-sg2hr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-n5824\" (UID: \"fa2eaa44-a12c-4056-9343-b0dbea9a57ff\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.331468 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.331440 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:27:58.480194 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.480171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824"] Jul 06 19:27:58.482895 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:27:58.482868 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2eaa44_a12c_4056_9343_b0dbea9a57ff.slice/crio-ce638c0f0969968e088c9746bbf9a3a5bb83eb4525eea0800f1ad63eea3e1f8b WatchSource:0}: Error finding container ce638c0f0969968e088c9746bbf9a3a5bb83eb4525eea0800f1ad63eea3e1f8b: Status 404 returned error can't find the container with id ce638c0f0969968e088c9746bbf9a3a5bb83eb4525eea0800f1ad63eea3e1f8b Jul 06 19:27:58.484233 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:58.484216 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:27:59.122660 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:59.122620 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" event={"ID":"fa2eaa44-a12c-4056-9343-b0dbea9a57ff","Type":"ContainerStarted","Data":"25a7869cf0884be112df9d4a6bf0bbb806e8582112bbd3526029f9395db82e28"} Jul 06 19:27:59.123035 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:59.122673 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" event={"ID":"fa2eaa44-a12c-4056-9343-b0dbea9a57ff","Type":"ContainerStarted","Data":"ce638c0f0969968e088c9746bbf9a3a5bb83eb4525eea0800f1ad63eea3e1f8b"} Jul 06 19:27:59.144877 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:59.144830 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" podStartSLOduration=1.971555162 podStartE2EDuration="2.144815825s" totalImagesPullingTime="173.260663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:59.14368931 +0000 UTC m=+990.673034335" watchObservedRunningTime="2026-07-06 19:27:59.144815825 +0000 UTC m=+990.674160849" Jul 06 19:27:59.922686 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:27:59.922654 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:28:07.308563 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.308529 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d"] Jul 06 19:28:07.312088 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.312073 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.314913 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.314894 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 19:28:07.328387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.328122 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d"] Jul 06 19:28:07.441890 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.441851 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.441890 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.441888 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c27686-3b9a-4df4-a4b4-70a8377b6679-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.442108 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.441920 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.442108 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.441993 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.442108 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.442089 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.442198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.442108 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rdp\" (UniqueName: \"kubernetes.io/projected/17c27686-3b9a-4df4-a4b4-70a8377b6679-kube-api-access-m7rdp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.542784 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542744 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.542784 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542786 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rdp\" (UniqueName: \"kubernetes.io/projected/17c27686-3b9a-4df4-a4b4-70a8377b6679-kube-api-access-m7rdp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543031 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542824 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543031 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c27686-3b9a-4df4-a4b4-70a8377b6679-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543031 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542895 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543031 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.542937 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543268 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.543242 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.543286 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.543413 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.543337 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.545174 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.545152 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c27686-3b9a-4df4-a4b4-70a8377b6679-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.545435 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.545419 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c27686-3b9a-4df4-a4b4-70a8377b6679-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.551966 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.551942 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rdp\" (UniqueName: \"kubernetes.io/projected/17c27686-3b9a-4df4-a4b4-70a8377b6679-kube-api-access-m7rdp\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d\" (UID: \"17c27686-3b9a-4df4-a4b4-70a8377b6679\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.621691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.621621 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:07.761191 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:07.761009 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d"] Jul 06 19:28:07.764132 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:28:07.764109 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17c27686_3b9a_4df4_a4b4_70a8377b6679.slice/crio-eb08310c23bff5ad10fec2308a6b6063afbc3827da6b5cff32abb075338f6ab9 WatchSource:0}: Error finding container eb08310c23bff5ad10fec2308a6b6063afbc3827da6b5cff32abb075338f6ab9: Status 404 returned error can't find the container with id eb08310c23bff5ad10fec2308a6b6063afbc3827da6b5cff32abb075338f6ab9 Jul 06 19:28:08.169249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.169215 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" event={"ID":"17c27686-3b9a-4df4-a4b4-70a8377b6679","Type":"ContainerStarted","Data":"3563d829e76edb01163726a68f7fc5409f5e9c5b33272c295315354c48168ace"} Jul 06 19:28:08.169249 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.169257 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" event={"ID":"17c27686-3b9a-4df4-a4b4-70a8377b6679","Type":"ContainerStarted","Data":"eb08310c23bff5ad10fec2308a6b6063afbc3827da6b5cff32abb075338f6ab9"} Jul 06 19:28:08.332339 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.332247 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:28:08.332339 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.332331 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:28:08.344893 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.344865 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:28:08.374966 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:08.374922 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" podStartSLOduration=1.204248572 podStartE2EDuration="1.374907288s" totalImagesPullingTime="170.658716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:08.206669356 +0000 UTC m=+999.736014381" watchObservedRunningTime="2026-07-06 19:28:08.374907288 +0000 UTC m=+999.904252291" Jul 06 19:28:09.185785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:09.185757 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-n5824" Jul 06 19:28:09.664928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:09.664843 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:28:17.622532 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:17.622489 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:17.623008 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:17.622549 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:17.635263 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:17.635231 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:28:18.228727 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:28:18.228696 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d" Jul 06 19:29:52.108464 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:29:52.108382 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:30:01.713856 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:30:01.713825 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:30:10.728819 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:30:10.728786 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:30:20.716395 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:30:20.716343 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:30:32.221727 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:30:32.221694 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:30:41.143026 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:30:41.142989 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:31:04.412287 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:04.412205 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:31:14.023812 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:14.023780 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:31:28.990460 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:28.990436 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:31:28.992312 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:28.992289 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:31:47.424297 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:47.424255 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:31:57.317464 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:31:57.317429 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:05.319036 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:05.318999 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:15.249795 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:15.249759 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:25.904544 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:25.904450 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:34.106815 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:34.106779 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:46.401025 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:46.400988 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:32:55.311266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:32:55.311227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:33:18.433669 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:33:18.433633 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:33:29.805273 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:33:29.805234 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:33:53.413334 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:33:53.413253 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:03.809999 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:03.809952 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:09.819893 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:09.819853 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:20.640694 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:20.640659 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:31.525166 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:31.525121 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:40.497702 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:40.497659 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:48.715537 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:48.715500 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:34:58.221787 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:34:58.221757 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:36:29.027759 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:36:29.027731 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:36:29.030028 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:36:29.030002 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:36:47.911313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:36:47.911277 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:36:55.347338 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:36:55.347240 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:09.007045 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:09.007008 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:16.707441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:16.707411 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:27.108849 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:27.108812 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:35.299337 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:35.299302 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:46.125848 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:46.125810 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:37:57.808539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:37:57.808500 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:38:07.018529 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:38:07.018496 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:38:17.306278 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:38:17.306242 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:38:27.908523 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:38:27.908438 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:38:37.201084 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:38:37.201050 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:01.303632 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:01.303601 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:07.910512 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:07.910475 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:18.818083 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:18.818047 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:29.822288 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:29.822253 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:44.807102 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:44.807069 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:39:57.926911 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:39:57.926811 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:08.135320 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:08.135288 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:20.818455 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:20.818425 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:27.515047 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:27.515009 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:36.125664 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:36.125630 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:49.916399 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:49.916368 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:40:59.540895 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:40:59.540862 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:41:08.421158 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:08.421120 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:41:19.225407 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:19.225373 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:41:24.417502 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:24.417419 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:41:29.060335 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:29.060305 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:41:29.064242 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:29.064220 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:41:33.436584 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:33.436551 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:41:44.718367 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:41:44.718316 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:01.166121 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:01.166089 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:12.105037 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:12.105002 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:22.109264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:22.109230 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:36.341266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:36.341227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:44.524027 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:44.523995 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:42:56.199811 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:42:56.199710 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:45:08.961279 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:45:08.961248 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:45:16.327444 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:45:16.327413 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:45:23.136126 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:45:23.136080 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:45:32.335416 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:45:32.335378 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:45:44.913709 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:45:44.913674 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:46:19.621884 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:19.621840 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:46:27.405396 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:27.405359 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:46:29.089601 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:29.089573 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:46:29.095313 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:29.095294 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:46:44.925980 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:44.925942 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:46:57.161977 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:46:57.161942 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:47:36.644913 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:47:36.644878 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:48:08.229345 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:48:08.229304 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:48:15.637446 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:48:15.637405 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:49:05.032572 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:05.032491 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:49:15.909615 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:15.909581 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:49:28.941184 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:28.941151 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:49:38.810839 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:38.810803 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:49:50.407597 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.407559 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:49:50.411223 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.411205 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.414264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.414240 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1ee26\"/\"kube-root-ca.crt\"" Jul 06 19:49:50.414264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.414262 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1ee26\"/\"default-dockercfg-kwmdl\"" Jul 06 19:49:50.414883 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.414868 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1ee26\"/\"openshift-service-ca.crt\"" Jul 06 19:49:50.415148 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.415135 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1ee26\"/\"test-model-f1ee26-kserve-self-signed-certs\"" Jul 06 19:49:50.425941 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.425916 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:49:50.426234 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426216 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.426298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426269 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.426298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426285 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.426429 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426408 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.426489 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426441 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fp8r\" (UniqueName: \"kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.426547 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.426537 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527545 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527513 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527545 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fp8r\" (UniqueName: \"kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527600 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527676 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.527802 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527707 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.528016 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.527978 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.528074 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.528040 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.528136 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.528119 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.530141 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.530118 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.530310 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.530292 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.544847 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.544819 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fp8r\" (UniqueName: \"kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r\") pod \"test-model-f1ee26-kserve-fd5d577c9-87tgc\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.720107 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.720071 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:49:50.861948 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:50.861924 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:49:50.864236 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:49:50.864211 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaae0ffd6_0a9d_4948_84e5_2709f384399f.slice/crio-065cd64897fef827d00e6f6ece1b561f7cd1bb5aa86aa24e9aa17c36239a3b85 WatchSource:0}: Error finding container 065cd64897fef827d00e6f6ece1b561f7cd1bb5aa86aa24e9aa17c36239a3b85: Status 404 returned error can't find the container with id 065cd64897fef827d00e6f6ece1b561f7cd1bb5aa86aa24e9aa17c36239a3b85 Jul 06 19:49:51.437967 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:51.437934 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" event={"ID":"aae0ffd6-0a9d-4948-84e5-2709f384399f","Type":"ContainerStarted","Data":"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0"} Jul 06 19:49:51.437967 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:51.437972 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" event={"ID":"aae0ffd6-0a9d-4948-84e5-2709f384399f","Type":"ContainerStarted","Data":"065cd64897fef827d00e6f6ece1b561f7cd1bb5aa86aa24e9aa17c36239a3b85"} Jul 06 19:49:51.463261 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:51.463210 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" podStartSLOduration=1.463192468 podStartE2EDuration="1.463192468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:51.460241986 +0000 UTC m=+2302.989587011" watchObservedRunningTime="2026-07-06 19:49:51.463192468 +0000 UTC m=+2302.992537492" Jul 06 19:49:53.825254 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:49:53.825210 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:50:00.720360 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:00.720314 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:00.720865 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:00.720408 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:00.733197 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:00.733173 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:01.498062 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:01.498034 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:13.194734 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:13.194704 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:50:16.633198 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:16.633161 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:50:28.215131 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.215026 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" podUID="aae0ffd6-0a9d-4948-84e5-2709f384399f" containerName="main" containerID="cri-o://4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0" gracePeriod=45 Jul 06 19:50:28.460967 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.460939 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:28.484804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484754 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.484804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484784 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.484928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484821 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.484928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484843 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.484928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484862 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.484928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.484886 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fp8r\" (UniqueName: \"kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r\") pod \"aae0ffd6-0a9d-4948-84e5-2709f384399f\" (UID: \"aae0ffd6-0a9d-4948-84e5-2709f384399f\") " Jul 06 19:50:28.485122 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.485044 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:28.485122 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.485076 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home" (OuterVolumeSpecName: "home") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:28.485229 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.485137 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache" (OuterVolumeSpecName: "model-cache") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:28.485229 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.485199 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-home\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.485229 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.485216 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-tmp-dir\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.487858 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.487809 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm" (OuterVolumeSpecName: "dshm") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:50:28.487977 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.487873 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:50:28.487977 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.487889 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r" (OuterVolumeSpecName: "kube-api-access-7fp8r") pod "aae0ffd6-0a9d-4948-84e5-2709f384399f" (UID: "aae0ffd6-0a9d-4948-84e5-2709f384399f"). InnerVolumeSpecName "kube-api-access-7fp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:28.586441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.586410 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-model-cache\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.586441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.586439 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fp8r\" (UniqueName: \"kubernetes.io/projected/aae0ffd6-0a9d-4948-84e5-2709f384399f-kube-api-access-7fp8r\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.586642 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.586451 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aae0ffd6-0a9d-4948-84e5-2709f384399f-dshm\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.586642 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.586463 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aae0ffd6-0a9d-4948-84e5-2709f384399f-tls-certs\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:50:28.615316 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.615274 2558 generic.go:356] "Generic (PLEG): container finished" podID="aae0ffd6-0a9d-4948-84e5-2709f384399f" containerID="4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0" exitCode=0 Jul 06 19:50:28.615478 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.615325 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" Jul 06 19:50:28.615478 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.615380 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" event={"ID":"aae0ffd6-0a9d-4948-84e5-2709f384399f","Type":"ContainerDied","Data":"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0"} Jul 06 19:50:28.615478 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.615442 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc" event={"ID":"aae0ffd6-0a9d-4948-84e5-2709f384399f","Type":"ContainerDied","Data":"065cd64897fef827d00e6f6ece1b561f7cd1bb5aa86aa24e9aa17c36239a3b85"} Jul 06 19:50:28.615478 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.615461 2558 scope.go:117] "RemoveContainer" containerID="4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0" Jul 06 19:50:28.624574 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.624559 2558 scope.go:117] "RemoveContainer" containerID="4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0" Jul 06 19:50:28.624809 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:50:28.624790 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0\": container with ID starting with 4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0 not found: ID does not exist" containerID="4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0" Jul 06 19:50:28.624857 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.624817 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0"} err="failed to get container status \"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0\": rpc error: code = NotFound desc = could not find container \"4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0\": container with ID starting with 4c24783b5481cfac1df125b0bfe3a6434019f3fe405c4564563160f566b1caf0 not found: ID does not exist" Jul 06 19:50:28.637509 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.637480 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:50:28.640971 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:28.640952 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f1ee26/test-model-f1ee26-kserve-fd5d577c9-87tgc"] Jul 06 19:50:29.018553 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:50:29.018518 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aae0ffd6-0a9d-4948-84e5-2709f384399f" path="/var/lib/kubelet/pods/aae0ffd6-0a9d-4948-84e5-2709f384399f/volumes" Jul 06 19:51:00.315501 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:00.315461 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:51:13.912190 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:13.912150 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:51:29.118615 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:29.118585 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:51:29.126428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:29.126405 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:51:30.620754 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:30.620722 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:51:41.201034 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:41.200998 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:51:54.313560 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:51:54.313484 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:52:07.250331 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:07.250290 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:52:12.634028 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.633993 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:52:12.634451 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.634384 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aae0ffd6-0a9d-4948-84e5-2709f384399f" containerName="main" Jul 06 19:52:12.634451 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.634396 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aae0ffd6-0a9d-4948-84e5-2709f384399f" containerName="main" Jul 06 19:52:12.634544 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.634458 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aae0ffd6-0a9d-4948-84e5-2709f384399f" containerName="main" Jul 06 19:52:12.637519 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.637503 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.642318 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.642291 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"kube-root-ca.crt\"" Jul 06 19:52:12.642479 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.642318 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"auth-test-model-2240fd-kserve-self-signed-certs\"" Jul 06 19:52:12.642479 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.642340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"default-dockercfg-64qxf\"" Jul 06 19:52:12.648364 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.648330 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"openshift-service-ca.crt\"" Jul 06 19:52:12.709373 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.709324 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:52:12.742410 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.742553 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742418 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.742553 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742462 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rz4\" (UniqueName: \"kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.742553 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742513 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.742695 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742585 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.742695 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.742603 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843521 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843641 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843557 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843715 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843669 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843780 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rz4\" (UniqueName: \"kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843780 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843750 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.843928 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843908 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.844065 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.843971 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.844065 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.844045 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.846078 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.846055 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.846257 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.846240 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.858959 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.858938 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rz4\" (UniqueName: \"kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4\") pod \"auth-test-model-2240fd-kserve-fdb649f79-kz8lt\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:12.947010 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:12.946983 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:13.088562 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:13.088538 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:52:13.090653 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:52:13.090630 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e1836ff_9420_480d_9436_535661123144.slice/crio-cf749aca4346224ae59af53a2fb012631a9bdd54eaf8148f0dd6600da333aa17 WatchSource:0}: Error finding container cf749aca4346224ae59af53a2fb012631a9bdd54eaf8148f0dd6600da333aa17: Status 404 returned error can't find the container with id cf749aca4346224ae59af53a2fb012631a9bdd54eaf8148f0dd6600da333aa17 Jul 06 19:52:13.119861 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:13.119838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" event={"ID":"7e1836ff-9420-480d-9436-535661123144","Type":"ContainerStarted","Data":"cf749aca4346224ae59af53a2fb012631a9bdd54eaf8148f0dd6600da333aa17"} Jul 06 19:52:14.127214 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:14.127182 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" event={"ID":"7e1836ff-9420-480d-9436-535661123144","Type":"ContainerStarted","Data":"82ac7168644f218e31dd58e32ca691c9cb5d5a273fe6665a7761a42e36ef7971"} Jul 06 19:52:14.148187 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:14.148143 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" podStartSLOduration=2.148126352 podStartE2EDuration="2.148126352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:52:14.146318215 +0000 UTC m=+2445.675663265" watchObservedRunningTime="2026-07-06 19:52:14.148126352 +0000 UTC m=+2445.677471377" Jul 06 19:52:15.430779 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:15.430736 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:52:22.947924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:22.947876 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:22.947924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:22.947933 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:22.960692 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:22.960665 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:23.182156 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:23.182129 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:52:32.301160 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:32.301125 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:52:44.826919 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:44.826885 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:52:57.721063 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:52:57.721018 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:53:13.502122 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:53:13.502087 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:53:28.608266 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:53:28.608185 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:53:44.996276 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:53:44.996246 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:53:58.708920 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:53:58.708886 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:54:14.320656 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:14.320624 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:54:27.698226 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:27.698191 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:54:30.773565 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:30.773530 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:54:35.283239 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.283209 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:54:35.286874 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.286858 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.290078 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.290053 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"sub-test-model-2240fd-kserve-self-signed-certs\"" Jul 06 19:54:35.300064 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.300040 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:54:35.342924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.342894 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.342924 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.342928 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.343119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.342955 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.343119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.343033 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.343119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.343068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.343119 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.343103 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvw4s\" (UniqueName: \"kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444377 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444552 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444391 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444552 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444552 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444460 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444552 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444516 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444712 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvw4s\" (UniqueName: \"kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444757 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444742 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444804 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444784 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.444961 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.444934 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.446916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.446895 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.447133 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.447115 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.453085 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.453063 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvw4s\" (UniqueName: \"kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s\") pod \"sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.597248 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.597161 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:35.736201 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.736177 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:54:35.738411 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:54:35.738379 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4863e67d_cbba_4b92_aa2f_f6716edeb58b.slice/crio-323cf9fbc9263be2e82209d3a68b9ee7426b97695fc706bccd9e99d6a4cd6e8c WatchSource:0}: Error finding container 323cf9fbc9263be2e82209d3a68b9ee7426b97695fc706bccd9e99d6a4cd6e8c: Status 404 returned error can't find the container with id 323cf9fbc9263be2e82209d3a68b9ee7426b97695fc706bccd9e99d6a4cd6e8c Jul 06 19:54:35.819894 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.819854 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" event={"ID":"4863e67d-cbba-4b92-aa2f-f6716edeb58b","Type":"ContainerStarted","Data":"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d"} Jul 06 19:54:35.820057 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:35.819904 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" event={"ID":"4863e67d-cbba-4b92-aa2f-f6716edeb58b","Type":"ContainerStarted","Data":"323cf9fbc9263be2e82209d3a68b9ee7426b97695fc706bccd9e99d6a4cd6e8c"} Jul 06 19:54:42.718861 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:42.718813 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" podUID="7e1836ff-9420-480d-9436-535661123144" containerName="main" containerID="cri-o://82ac7168644f218e31dd58e32ca691c9cb5d5a273fe6665a7761a42e36ef7971" gracePeriod=45 Jul 06 19:54:42.854881 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:42.854851 2558 generic.go:356] "Generic (PLEG): container finished" podID="7e1836ff-9420-480d-9436-535661123144" containerID="82ac7168644f218e31dd58e32ca691c9cb5d5a273fe6665a7761a42e36ef7971" exitCode=0 Jul 06 19:54:42.855038 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:42.854944 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" event={"ID":"7e1836ff-9420-480d-9436-535661123144","Type":"ContainerDied","Data":"82ac7168644f218e31dd58e32ca691c9cb5d5a273fe6665a7761a42e36ef7971"} Jul 06 19:54:42.980601 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:42.980542 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:54:42.997538 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:42.997485 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" podStartSLOduration=7.997469377 podStartE2EDuration="7.997469377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:54:35.842592665 +0000 UTC m=+2587.371937690" watchObservedRunningTime="2026-07-06 19:54:42.997469377 +0000 UTC m=+2594.526814440" Jul 06 19:54:43.117627 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117592 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.117792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117640 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.117792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117690 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.117792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117740 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.117792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117773 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.118011 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117798 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6rz4\" (UniqueName: \"kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4\") pod \"7e1836ff-9420-480d-9436-535661123144\" (UID: \"7e1836ff-9420-480d-9436-535661123144\") " Jul 06 19:54:43.118011 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.117877 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.118109 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.118087 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.118109 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.118097 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home" (OuterVolumeSpecName: "home") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.118219 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.118115 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-model-cache\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.119917 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.119882 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm" (OuterVolumeSpecName: "dshm") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:54:43.120033 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.119969 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:54:43.120033 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.120021 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4" (OuterVolumeSpecName: "kube-api-access-m6rz4") pod "7e1836ff-9420-480d-9436-535661123144" (UID: "7e1836ff-9420-480d-9436-535661123144"). InnerVolumeSpecName "kube-api-access-m6rz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:54:43.219235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.219203 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-dshm\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.219235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.219229 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e1836ff-9420-480d-9436-535661123144-tls-certs\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.219235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.219240 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-tmp-dir\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.219235 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.219247 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e1836ff-9420-480d-9436-535661123144-home\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.219520 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.219256 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6rz4\" (UniqueName: \"kubernetes.io/projected/7e1836ff-9420-480d-9436-535661123144-kube-api-access-m6rz4\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:54:43.860616 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.860583 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" Jul 06 19:54:43.861098 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.860626 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt" event={"ID":"7e1836ff-9420-480d-9436-535661123144","Type":"ContainerDied","Data":"cf749aca4346224ae59af53a2fb012631a9bdd54eaf8148f0dd6600da333aa17"} Jul 06 19:54:43.861098 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.860676 2558 scope.go:117] "RemoveContainer" containerID="82ac7168644f218e31dd58e32ca691c9cb5d5a273fe6665a7761a42e36ef7971" Jul 06 19:54:43.883253 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.883226 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:54:43.887443 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:43.887417 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/auth-test-model-2240fd-kserve-fdb649f79-kz8lt"] Jul 06 19:54:44.203278 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:44.203246 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:54:45.017391 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:45.017335 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1836ff-9420-480d-9436-535661123144" path="/var/lib/kubelet/pods/7e1836ff-9420-480d-9436-535661123144/volumes" Jul 06 19:54:45.598296 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:45.598250 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:45.598577 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:45.598328 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:45.610982 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:45.610959 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:54:45.882514 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:54:45.882428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:55:13.220913 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:55:13.220877 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:55:33.212792 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:55:33.212754 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:55:50.204582 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:55:50.204547 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:55:56.282849 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:55:56.282817 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:56:06.711401 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:06.711365 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:56:11.310363 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.310315 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" podUID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" containerName="main" containerID="cri-o://4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d" gracePeriod=45 Jul 06 19:56:11.561627 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.561566 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:56:11.614796 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614763 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.614965 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614836 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.614965 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614875 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.614965 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614918 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.614965 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614944 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvw4s\" (UniqueName: \"kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.615183 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.614988 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir\") pod \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\" (UID: \"4863e67d-cbba-4b92-aa2f-f6716edeb58b\") " Jul 06 19:56:11.615183 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615055 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home" (OuterVolumeSpecName: "home") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:11.615283 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615203 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:11.615283 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615253 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:11.615387 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615345 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-home\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:11.615440 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615388 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-model-cache\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:11.615440 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.615403 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tmp-dir\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:11.617284 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.617264 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s" (OuterVolumeSpecName: "kube-api-access-zvw4s") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "kube-api-access-zvw4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:56:11.617548 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.617534 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm" (OuterVolumeSpecName: "dshm") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:56:11.617595 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.617557 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4863e67d-cbba-4b92-aa2f-f6716edeb58b" (UID: "4863e67d-cbba-4b92-aa2f-f6716edeb58b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:56:11.716163 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.716137 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4863e67d-cbba-4b92-aa2f-f6716edeb58b-dshm\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:11.716163 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.716160 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4863e67d-cbba-4b92-aa2f-f6716edeb58b-tls-certs\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:11.716339 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:11.716173 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvw4s\" (UniqueName: \"kubernetes.io/projected/4863e67d-cbba-4b92-aa2f-f6716edeb58b-kube-api-access-zvw4s\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:56:12.298888 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.298850 2558 generic.go:356] "Generic (PLEG): container finished" podID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" containerID="4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d" exitCode=0 Jul 06 19:56:12.299046 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.298895 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" Jul 06 19:56:12.299046 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.298927 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" event={"ID":"4863e67d-cbba-4b92-aa2f-f6716edeb58b","Type":"ContainerDied","Data":"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d"} Jul 06 19:56:12.299046 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.298970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj" event={"ID":"4863e67d-cbba-4b92-aa2f-f6716edeb58b","Type":"ContainerDied","Data":"323cf9fbc9263be2e82209d3a68b9ee7426b97695fc706bccd9e99d6a4cd6e8c"} Jul 06 19:56:12.299046 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.298987 2558 scope.go:117] "RemoveContainer" containerID="4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d" Jul 06 19:56:12.308333 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.308316 2558 scope.go:117] "RemoveContainer" containerID="4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d" Jul 06 19:56:12.308588 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:56:12.308571 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d\": container with ID starting with 4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d not found: ID does not exist" containerID="4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d" Jul 06 19:56:12.308657 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.308595 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d"} err="failed to get container status \"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d\": rpc error: code = NotFound desc = could not find container \"4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d\": container with ID starting with 4c2f4892471f2e8d566ae1f8dd8c3f3dbe8e266cf83699543711efd243d2510d not found: ID does not exist" Jul 06 19:56:12.319785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.319755 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:56:12.326006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:12.325976 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/sub-test-model-2240fd-kserve-647b6bcbf9-qm7nj"] Jul 06 19:56:13.022325 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:13.022291 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" path="/var/lib/kubelet/pods/4863e67d-cbba-4b92-aa2f-f6716edeb58b/volumes" Jul 06 19:56:22.615410 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615319 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:56:22.615785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615699 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" containerName="main" Jul 06 19:56:22.615785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615710 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" containerName="main" Jul 06 19:56:22.615785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615754 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1836ff-9420-480d-9436-535661123144" containerName="main" Jul 06 19:56:22.615785 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615765 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1836ff-9420-480d-9436-535661123144" containerName="main" Jul 06 19:56:22.615916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615824 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1836ff-9420-480d-9436-535661123144" containerName="main" Jul 06 19:56:22.615916 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.615836 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4863e67d-cbba-4b92-aa2f-f6716edeb58b" containerName="main" Jul 06 19:56:22.620052 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.620033 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.624428 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.624395 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"kube-root-ca.crt\"" Jul 06 19:56:22.624541 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.624395 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"default-dockercfg-64qxf\"" Jul 06 19:56:22.624541 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.624419 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"openshift-service-ca.crt\"" Jul 06 19:56:22.624943 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.624929 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2240fd\"/\"rate-test-model-2240fd-kserve-self-signed-certs\"" Jul 06 19:56:22.636487 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.636460 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:56:22.720486 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720454 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.720633 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720497 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.720633 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720561 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqnpj\" (UniqueName: \"kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.720633 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720578 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.720633 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720593 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.720837 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.720692 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.821906 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.821873 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.821913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.821962 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqnpj\" (UniqueName: \"kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.821993 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.822009 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.822033 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822626 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.822417 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822626 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.822468 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.822626 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.822564 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.824435 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.824417 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.824707 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.824687 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.839851 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.839825 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqnpj\" (UniqueName: \"kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj\") pod \"rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:22.929212 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:22.929181 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:23.072677 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:23.072653 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:56:23.075315 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:56:23.075293 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5b0f4bc_e76a_486b_b745_40f3bda0419a.slice/crio-35bbf45831dcce0c298f824c6caebb2933441120d9d164fece4ecd1d4f7bfdca WatchSource:0}: Error finding container 35bbf45831dcce0c298f824c6caebb2933441120d9d164fece4ecd1d4f7bfdca: Status 404 returned error can't find the container with id 35bbf45831dcce0c298f824c6caebb2933441120d9d164fece4ecd1d4f7bfdca Jul 06 19:56:23.360298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:23.360194 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" event={"ID":"a5b0f4bc-e76a-486b-b745-40f3bda0419a","Type":"ContainerStarted","Data":"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555"} Jul 06 19:56:23.360298 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:23.360241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" event={"ID":"a5b0f4bc-e76a-486b-b745-40f3bda0419a","Type":"ContainerStarted","Data":"35bbf45831dcce0c298f824c6caebb2933441120d9d164fece4ecd1d4f7bfdca"} Jul 06 19:56:23.388141 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:23.388074 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" podStartSLOduration=1.3880593509999999 podStartE2EDuration="1.388059351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:56:23.386049624 +0000 UTC m=+2694.915394656" watchObservedRunningTime="2026-07-06 19:56:23.388059351 +0000 UTC m=+2694.917404429" Jul 06 19:56:26.407278 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:26.407242 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:56:29.152528 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:29.152501 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:56:29.159566 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:29.159544 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gzcfs_75355291-7dac-483c-ac38-bf786fe91684/ovn-acl-logging/0.log" Jul 06 19:56:32.930188 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:32.930134 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:32.930188 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:32.930202 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:32.942825 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:32.942794 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:33.417880 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:33.417795 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:56:51.017782 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:56:51.017749 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:57:17.246309 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:17.246274 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:57:29.813344 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:29.813304 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:57:37.832738 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:37.832699 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:57:44.270670 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:44.270635 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:57:52.853263 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:52.853169 2558 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" podUID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" containerName="main" containerID="cri-o://8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555" gracePeriod=45 Jul 06 19:57:53.107619 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.107533 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:57:53.173625 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173598 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqnpj\" (UniqueName: \"kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.173814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173645 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.173814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173677 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.173814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173710 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.173814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173733 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.173814 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.173765 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache\") pod \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\" (UID: \"a5b0f4bc-e76a-486b-b745-40f3bda0419a\") " Jul 06 19:57:53.174104 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174023 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home" (OuterVolumeSpecName: "home") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:53.174104 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174041 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:53.174104 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174082 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:53.174264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174237 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-home\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.174264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174249 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tmp-dir\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.174264 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.174257 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-model-cache\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.175910 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.175888 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm" (OuterVolumeSpecName: "dshm") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:57:53.176006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.175944 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:57:53.176006 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.175946 2558 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj" (OuterVolumeSpecName: "kube-api-access-bqnpj") pod "a5b0f4bc-e76a-486b-b745-40f3bda0419a" (UID: "a5b0f4bc-e76a-486b-b745-40f3bda0419a"). InnerVolumeSpecName "kube-api-access-bqnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:57:53.275237 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.275202 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b0f4bc-e76a-486b-b745-40f3bda0419a-dshm\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.275237 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.275231 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b0f4bc-e76a-486b-b745-40f3bda0419a-tls-certs\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.275237 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.275243 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqnpj\" (UniqueName: \"kubernetes.io/projected/a5b0f4bc-e76a-486b-b745-40f3bda0419a-kube-api-access-bqnpj\") on node \"ip-10-0-143-63.ec2.internal\" DevicePath \"\"" Jul 06 19:57:53.793935 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.793897 2558 generic.go:356] "Generic (PLEG): container finished" podID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" containerID="8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555" exitCode=0 Jul 06 19:57:53.794142 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.793939 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" event={"ID":"a5b0f4bc-e76a-486b-b745-40f3bda0419a","Type":"ContainerDied","Data":"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555"} Jul 06 19:57:53.794142 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.793960 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" Jul 06 19:57:53.794142 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.793992 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd" event={"ID":"a5b0f4bc-e76a-486b-b745-40f3bda0419a","Type":"ContainerDied","Data":"35bbf45831dcce0c298f824c6caebb2933441120d9d164fece4ecd1d4f7bfdca"} Jul 06 19:57:53.794142 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.794013 2558 scope.go:117] "RemoveContainer" containerID="8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555" Jul 06 19:57:53.803247 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.803229 2558 scope.go:117] "RemoveContainer" containerID="8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555" Jul 06 19:57:53.803585 ip-10-0-143-63 kubenswrapper[2558]: E0706 19:57:53.803564 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555\": container with ID starting with 8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555 not found: ID does not exist" containerID="8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555" Jul 06 19:57:53.803668 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.803591 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555"} err="failed to get container status \"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555\": rpc error: code = NotFound desc = could not find container \"8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555\": container with ID starting with 8534562900d1ec787127f7f5b3938f3e35c02a6eb2fc300e6b05b51948465555 not found: ID does not exist" Jul 06 19:57:53.819132 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.819107 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:57:53.828324 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:53.828299 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2240fd/rate-test-model-2240fd-kserve-7fbbb9f87f-mb8zd"] Jul 06 19:57:55.018182 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:57:55.018150 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" path="/var/lib/kubelet/pods/a5b0f4bc-e76a-486b-b745-40f3bda0419a/volumes" Jul 06 19:58:26.309479 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:26.309448 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-fcxhd"] Jul 06 19:58:37.000868 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:37.000838 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-j4bt9_b127174a-ba46-42a3-b0ca-190ed348baae/copy-manifests/0.log" Jul 06 19:58:37.013907 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:37.013887 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-j4bt9_b127174a-ba46-42a3-b0ca-190ed348baae/manager/0.log" Jul 06 19:58:39.011470 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:39.011436 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nxlcd_ba70de49-01a9-40ec-81af-9a0dc112bfc2/kuadrant-console-plugin/0.log" Jul 06 19:58:39.392491 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:39.392409 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcxhd_cf6b915b-45d1-4fe8-92d4-b3f190f924cd/limitador/0.log" Jul 06 19:58:40.285023 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:40.284995 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-zbhw9_c99602cc-8c62-49d9-bb00-11c6c336167c/kube-auth-proxy/0.log" Jul 06 19:58:41.164806 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:41.164780 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-92kwk_c3913cab-6dbd-44ed-99e5-1679ece01530/main/0.log" Jul 06 19:58:41.648326 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:41.648291 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-n5824_fa2eaa44-a12c-4056-9343-b0dbea9a57ff/main/0.log" Jul 06 19:58:41.773765 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:41.773732 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-64b2d_17c27686-3b9a-4df4-a4b4-70a8377b6679/main/0.log" Jul 06 19:58:49.224768 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:49.224743 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x8h8p_61c0fbb0-8a79-474c-a770-2c14e44043b0/global-pull-secret-syncer/0.log" Jul 06 19:58:49.298269 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:49.298243 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ghvvr_4295aa48-adaf-4a8f-b821-38ff2f059993/konnectivity-agent/0.log" Jul 06 19:58:49.419017 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:49.418984 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-63.ec2.internal_3312f4e0b6092b33b85bb20010305980/haproxy/0.log" Jul 06 19:58:54.377187 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:54.377086 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nxlcd_ba70de49-01a9-40ec-81af-9a0dc112bfc2/kuadrant-console-plugin/0.log" Jul 06 19:58:54.665807 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:54.665722 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-fcxhd_cf6b915b-45d1-4fe8-92d4-b3f190f924cd/limitador/0.log" Jul 06 19:58:56.562714 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:56.562636 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-rnz9c_3528c111-3fde-4c3a-921c-744cfc1c4ade/cluster-monitoring-operator/0.log" Jul 06 19:58:56.664503 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:56.664472 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-546989849d-wvpt5_4e12011d-76e5-4afe-8b98-62c13720630f/metrics-server/0.log" Jul 06 19:58:56.916973 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:56.916888 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjgpq_4725c635-a8dc-41b7-8ddd-ee75b0e25ba6/node-exporter/0.log" Jul 06 19:58:56.943762 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:56.943734 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjgpq_4725c635-a8dc-41b7-8ddd-ee75b0e25ba6/kube-rbac-proxy/0.log" Jul 06 19:58:56.973281 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:56.973252 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sjgpq_4725c635-a8dc-41b7-8ddd-ee75b0e25ba6/init-textfile/0.log" Jul 06 19:58:57.011113 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.011088 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7vshg_8dbd6613-0437-4f52-b622-9b3ae215edec/kube-rbac-proxy-main/0.log" Jul 06 19:58:57.047543 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.047513 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7vshg_8dbd6613-0437-4f52-b622-9b3ae215edec/kube-rbac-proxy-self/0.log" Jul 06 19:58:57.072558 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.072529 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7vshg_8dbd6613-0437-4f52-b622-9b3ae215edec/openshift-state-metrics/0.log" Jul 06 19:58:57.113898 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.113859 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/prometheus/0.log" Jul 06 19:58:57.142170 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.142144 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/config-reloader/0.log" Jul 06 19:58:57.167807 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.167743 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/thanos-sidecar/0.log" Jul 06 19:58:57.191441 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.191421 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/kube-rbac-proxy-web/0.log" Jul 06 19:58:57.222467 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.222445 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/kube-rbac-proxy/0.log" Jul 06 19:58:57.246529 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.246504 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/kube-rbac-proxy-thanos/0.log" Jul 06 19:58:57.269809 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.269790 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5697d533-601e-40ba-881b-117dbc9ed917/init-config-reloader/0.log" Jul 06 19:58:57.299273 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.299252 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-7sthd_b373003a-544d-466f-a966-37728caa365f/prometheus-operator/0.log" Jul 06 19:58:57.320466 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.320443 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-7sthd_b373003a-544d-466f-a966-37728caa365f/kube-rbac-proxy/0.log" Jul 06 19:58:57.383308 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.383283 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-6vxwf_b197e777-3e93-4497-a0e6-50e7cd95708e/prometheus-operator-admission-webhook/0.log" Jul 06 19:58:57.519129 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.519103 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/thanos-query/0.log" Jul 06 19:58:57.543480 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.543453 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/kube-rbac-proxy-web/0.log" Jul 06 19:58:57.568830 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.568808 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/kube-rbac-proxy/0.log" Jul 06 19:58:57.594078 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.594055 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/prom-label-proxy/0.log" Jul 06 19:58:57.620586 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.620564 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/kube-rbac-proxy-rules/0.log" Jul 06 19:58:57.653644 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:57.653609 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54546fdfc-tpwf5_57605514-10be-447f-ac0b-00b057799d50/kube-rbac-proxy-metrics/0.log" Jul 06 19:58:58.906755 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:58.906724 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-4rglq_7be2f178-730b-4a7e-8a12-a163c0649821/networking-console-plugin/0.log" Jul 06 19:58:59.109591 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.109548 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph"] Jul 06 19:58:59.110061 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.110043 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" containerName="main" Jul 06 19:58:59.110135 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.110065 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" containerName="main" Jul 06 19:58:59.110189 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.110177 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b0f4bc-e76a-486b-b745-40f3bda0419a" containerName="main" Jul 06 19:58:59.113549 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.113526 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.116289 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.116270 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"openshift-service-ca.crt\"" Jul 06 19:58:59.116412 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.116286 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq5kf\"/\"kube-root-ca.crt\"" Jul 06 19:58:59.117035 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.117022 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq5kf\"/\"default-dockercfg-xrlgm\"" Jul 06 19:58:59.124703 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.124677 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph"] Jul 06 19:58:59.178094 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.178068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-proc\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.178221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.178097 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-lib-modules\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.178221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.178177 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-podres\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.178221 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.178211 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbgk\" (UniqueName: \"kubernetes.io/projected/ef194b58-ea17-46d4-a5d5-e81c59c17030-kube-api-access-vlbgk\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.178321 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.178286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-sys\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279535 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-podres\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279542 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbgk\" (UniqueName: \"kubernetes.io/projected/ef194b58-ea17-46d4-a5d5-e81c59c17030-kube-api-access-vlbgk\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-sys\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279630 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-proc\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279654 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-lib-modules\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279691 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279658 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-podres\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279885 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279699 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-proc\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279885 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279701 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-sys\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.279885 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.279778 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef194b58-ea17-46d4-a5d5-e81c59c17030-lib-modules\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.288211 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.288182 2558 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbgk\" (UniqueName: \"kubernetes.io/projected/ef194b58-ea17-46d4-a5d5-e81c59c17030-kube-api-access-vlbgk\") pod \"perf-node-gather-daemonset-fdvph\" (UID: \"ef194b58-ea17-46d4-a5d5-e81c59c17030\") " pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.424539 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.424512 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:58:59.563769 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:58:59.563744 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph"] Jul 06 19:58:59.565886 ip-10-0-143-63 kubenswrapper[2558]: W0706 19:58:59.565860 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef194b58_ea17_46d4_a5d5_e81c59c17030.slice/crio-8c725b416a61f2084f34d121dfc677d1f9c19d2b97de23cc58e954d28169774c WatchSource:0}: Error finding container 8c725b416a61f2084f34d121dfc677d1f9c19d2b97de23cc58e954d28169774c: Status 404 returned error can't find the container with id 8c725b416a61f2084f34d121dfc677d1f9c19d2b97de23cc58e954d28169774c Jul 06 19:59:00.112747 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.112715 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" event={"ID":"ef194b58-ea17-46d4-a5d5-e81c59c17030","Type":"ContainerStarted","Data":"0d7375b032ecdd291102a791fe408c8a3e5043712bdf3e9f9f5e40953a9519ec"} Jul 06 19:59:00.113137 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.112769 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" event={"ID":"ef194b58-ea17-46d4-a5d5-e81c59c17030","Type":"ContainerStarted","Data":"8c725b416a61f2084f34d121dfc677d1f9c19d2b97de23cc58e954d28169774c"} Jul 06 19:59:00.113137 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.112861 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:59:00.116308 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.116282 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-786f59bffc-sxfkt_ca877ba2-25a7-4844-9fff-cce512f3a830/console/0.log" Jul 06 19:59:00.131851 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.131806 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" podStartSLOduration=1.131792019 podStartE2EDuration="1.131792019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:59:00.129455827 +0000 UTC m=+2851.658800871" watchObservedRunningTime="2026-07-06 19:59:00.131792019 +0000 UTC m=+2851.661137044" Jul 06 19:59:00.154835 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.154812 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-knrdz_b9120fd3-d5a2-46ca-af9f-d75952e2dad2/download-server/0.log" Jul 06 19:59:00.734872 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:00.734843 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-vtrnc_2e8cb9c4-b83f-4ff7-80d9-2485abe1096b/volume-data-source-validator/0.log" Jul 06 19:59:01.879049 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:01.879022 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tckbk_c0d423a6-8e0c-44a4-a3e2-4a8a76e80633/dns/0.log" Jul 06 19:59:01.901299 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:01.901279 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tckbk_c0d423a6-8e0c-44a4-a3e2-4a8a76e80633/kube-rbac-proxy/0.log" Jul 06 19:59:01.952662 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:01.952631 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7jbm_3b192ffc-504c-4231-9c5a-efa16fdac630/dns-node-resolver/0.log" Jul 06 19:59:02.672946 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:02.672915 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5qbrh_3e1ddb84-c2f2-42ce-9438-e2b4bf546381/node-ca/0.log" Jul 06 19:59:03.830529 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:03.830499 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-zbhw9_c99602cc-8c62-49d9-bb00-11c6c336167c/kube-auth-proxy/0.log" Jul 06 19:59:04.642179 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:04.642150 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zn82_3071cbb5-ad05-492b-a327-1cd14a91e3c4/serve-healthcheck-canary/0.log" Jul 06 19:59:05.348179 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:05.348147 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fxmgm_60123861-df62-4e28-8192-5ec05132a8d8/kube-rbac-proxy/0.log" Jul 06 19:59:05.370608 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:05.370585 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fxmgm_60123861-df62-4e28-8192-5ec05132a8d8/exporter/0.log" Jul 06 19:59:05.395581 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:05.395557 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fxmgm_60123861-df62-4e28-8192-5ec05132a8d8/extractor/0.log" Jul 06 19:59:06.128829 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:06.128802 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq5kf/perf-node-gather-daemonset-fdvph" Jul 06 19:59:08.448012 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:08.447984 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-j4bt9_b127174a-ba46-42a3-b0ca-190ed348baae/manager/0.log" Jul 06 19:59:08.478686 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:08.478661 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f978bf57-j4bt9_b127174a-ba46-42a3-b0ca-190ed348baae/copy-manifests/0.log" Jul 06 19:59:09.611868 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:09.611837 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-zbhw9_c99602cc-8c62-49d9-bb00-11c6c336167c/kube-auth-proxy/0.log" Jul 06 19:59:10.060860 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:10.060829 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-665d858957-gw8n8_d036056a-6067-4163-ab3c-af4a384496ff/manager/0.log" Jul 06 19:59:11.151129 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:11.151078 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fffdf4997-zbhw9_c99602cc-8c62-49d9-bb00-11c6c336167c/kube-auth-proxy/0.log" Jul 06 19:59:14.908071 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:14.908042 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-ldjm2_8801a073-e65e-40ef-83f6-4dd35c3fdf5c/migrator/0.log" Jul 06 19:59:14.949686 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:14.949660 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-ldjm2_8801a073-e65e-40ef-83f6-4dd35c3fdf5c/graceful-termination/0.log" Jul 06 19:59:15.389807 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:15.389758 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-98lq4_a54efb5e-c0ac-4ab6-99c4-0d833ab26d30/kube-storage-version-migrator-operator/1.log" Jul 06 19:59:15.390466 ip-10-0-143-63 kubenswrapper[2558]: I0706 19:59:15.390450 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-98lq4_a54efb5e-c0ac-4ab6-99c4-0d833ab26d30/kube-storage-version-migrator-operator/0.log"