Jul 07 16:21:02.006687 ip-10-0-142-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 16:21:02.006700 ip-10-0-142-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 16:21:02.006709 ip-10-0-142-74 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 16:21:02.007051 ip-10-0-142-74 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 16:21:12.084849 ip-10-0-142-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 16:21:12.084867 ip-10-0-142-74 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fbfb915f347a480497c1246d3bc6e21c -- Jul 07 16:23:15.575097 ip-10-0-142-74 systemd[1]: Starting Kubernetes Kubelet... Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.957671 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964074 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964085 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964088 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964092 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964096 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964099 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964101 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:15.976991 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964104 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964107 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964110 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964113 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964115 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964118 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964121 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964124 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964127 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964130 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964133 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964135 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964138 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964140 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964147 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964150 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964153 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964155 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964158 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964161 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.010829 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964163 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964166 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964169 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964171 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964174 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964177 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964179 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964182 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964184 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964187 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964189 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964192 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964195 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964197 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964200 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964202 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964205 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964207 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964210 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964213 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.012121 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964217 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964219 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964222 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964224 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964227 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964229 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964233 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964237 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964240 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964243 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964246 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964249 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964251 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964254 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964257 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964260 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964262 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964266 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964268 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.012984 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964271 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964273 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964276 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964278 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964281 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964283 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964286 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964289 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964292 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964295 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964297 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964300 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964302 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964305 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964307 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964310 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964312 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964315 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964317 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964319 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.013879 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964322 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964326 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964330 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964333 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964335 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964339 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964343 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964346 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964348 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964351 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964353 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964356 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.964358 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965033 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965040 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965043 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965045 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965048 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965051 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.014753 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965054 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965057 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965060 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965062 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965065 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965067 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965070 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965072 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965075 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965078 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965080 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965083 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965086 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965088 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965094 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965097 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965100 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965102 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965105 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.015627 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965109 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965114 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965116 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965120 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965123 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965125 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965128 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965130 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965133 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965136 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965138 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965141 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965143 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965147 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965151 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965153 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965158 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965162 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965179 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.016502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965184 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965187 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965190 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965193 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965196 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965199 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965201 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965204 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965208 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965210 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965213 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965216 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965218 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965221 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965223 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965226 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965228 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965231 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965233 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965236 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.017374 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965238 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965241 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965243 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965246 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965249 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965252 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965256 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965259 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965261 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965264 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965266 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965268 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965271 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965274 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965277 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965279 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965282 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965284 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965287 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965289 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.018251 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965293 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965296 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965298 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965301 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965304 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965306 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965309 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965311 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965314 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965316 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965319 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965321 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965324 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965326 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.965328 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968034 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968046 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968050 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968054 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968057 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.019136 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968061 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968065 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968068 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968070 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968073 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968076 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968078 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968081 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968083 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968086 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968088 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968091 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968094 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968097 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968099 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968102 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968104 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968107 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968110 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968112 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.020010 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968115 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968117 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968120 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968123 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968125 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968128 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968130 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968134 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968136 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968139 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968141 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968144 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968146 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968149 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968151 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968154 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968156 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968158 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968161 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968163 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.021091 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968166 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968169 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968172 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968174 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968177 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968180 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968182 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968185 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968190 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968196 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968199 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968203 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968205 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968209 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968213 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968216 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968219 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968222 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.022194 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968224 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968227 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968229 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968232 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968235 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968237 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968240 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968242 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968245 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968248 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968250 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968252 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968256 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968259 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968261 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968264 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968266 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968269 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968272 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968281 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.023073 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968283 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968286 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968289 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968293 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968296 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968299 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968302 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968304 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968306 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968309 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968312 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968314 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968317 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968319 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968322 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.968324 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968393 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968400 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968406 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968410 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968414 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 16:23:16.024147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968417 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968421 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968425 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968429 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968431 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968434 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968438 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968440 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968443 2569 flags.go:64] FLAG: --cgroup-root="" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968446 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968449 2569 flags.go:64] FLAG: --client-ca-file="" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968458 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968461 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968468 2569 flags.go:64] FLAG: --cluster-domain="" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968474 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968477 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968480 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968483 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968487 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968490 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968493 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968496 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968499 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968502 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 16:23:16.025043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968505 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968508 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968510 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968514 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968517 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968520 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968523 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968526 2569 flags.go:64] FLAG: --enable-server="true" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968528 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968535 2569 flags.go:64] FLAG: --event-burst="100" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968537 2569 flags.go:64] FLAG: --event-qps="50" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968540 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968543 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968546 2569 flags.go:64] FLAG: --eviction-hard="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968549 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968552 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968555 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968557 2569 flags.go:64] FLAG: --eviction-soft="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968560 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968563 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968573 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968576 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968581 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968584 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968587 2569 flags.go:64] FLAG: --feature-gates="" Jul 07 16:23:16.026150 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968590 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968593 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968596 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968599 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968602 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968605 2569 flags.go:64] FLAG: --help="false" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968608 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968611 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968614 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968616 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968620 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968623 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968626 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968628 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968631 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968633 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968636 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968639 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968642 2569 flags.go:64] FLAG: --kube-reserved="" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968645 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968647 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968650 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968653 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968655 2569 flags.go:64] FLAG: --lock-file="" Jul 07 16:23:16.027455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968658 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968660 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968664 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968669 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968677 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968682 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968685 2569 flags.go:64] FLAG: --logging-format="text" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968688 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968691 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968694 2569 flags.go:64] FLAG: --manifest-url="" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968697 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968701 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968705 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968708 2569 flags.go:64] FLAG: --max-pods="110" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968711 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968714 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968717 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968720 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968722 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968725 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968728 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968734 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968737 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968740 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 16:23:16.028542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968743 2569 flags.go:64] FLAG: --pod-cidr="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968746 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968751 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968754 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968757 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968759 2569 flags.go:64] FLAG: --port="10250" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968763 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968765 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02cbafc756a074130" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968768 2569 flags.go:64] FLAG: --qos-reserved="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968771 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968774 2569 flags.go:64] FLAG: --register-node="true" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968776 2569 flags.go:64] FLAG: --register-with-taints="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968780 2569 flags.go:64] FLAG: --registry-burst="10" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968789 2569 flags.go:64] FLAG: --registry-qps="5" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968793 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968796 2569 flags.go:64] FLAG: --reserved-memory="" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968800 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968803 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968805 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968808 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968811 2569 flags.go:64] FLAG: --runonce="false" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968814 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968816 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968819 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968822 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 16:23:16.029632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968825 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968828 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968846 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968850 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968852 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968855 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968858 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968861 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968864 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968866 2569 flags.go:64] FLAG: --system-cgroups="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968869 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968874 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968877 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968880 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968887 2569 flags.go:64] FLAG: --tls-min-version="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968890 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968893 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968895 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968898 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968901 2569 flags.go:64] FLAG: --v="2" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968906 2569 flags.go:64] FLAG: --version="false" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968912 2569 flags.go:64] FLAG: --vmodule="" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968916 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.968919 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969023 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.034195 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969026 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969030 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969033 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969036 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969038 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969041 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969043 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969047 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969051 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969053 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969056 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969058 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969062 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969066 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969069 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969072 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969075 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969077 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969080 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.035304 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969083 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969086 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969088 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969091 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969093 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969096 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969098 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969101 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969105 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969109 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969116 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969119 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969121 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969124 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969127 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969129 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969132 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969134 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969137 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969139 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.245816 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969142 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.068288 ip-10-0-142-74 systemd[1]: Started Kubernetes Kubelet. Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969144 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969147 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969149 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969152 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969154 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969157 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969159 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969162 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969165 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969167 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969170 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969172 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969175 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969177 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969180 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969182 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969184 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969187 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.260048 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969189 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969194 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969198 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969201 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969204 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969207 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969210 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969212 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969215 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969218 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969220 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969223 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969225 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969228 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969231 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969233 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969236 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969238 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969241 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969243 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.261309 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969246 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969248 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969251 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969253 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969256 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969258 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969261 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969263 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969266 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969268 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969270 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969273 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969275 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969279 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969283 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969286 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969288 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969292 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969296 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.262538 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.969298 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.969976 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.976308 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.976323 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.977609 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.977619 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977676 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977681 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977684 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977687 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977690 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977694 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977696 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977699 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.263502 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977702 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977704 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977708 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977710 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977713 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977716 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977718 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977721 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977723 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977726 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977728 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977731 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977734 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.977737 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978539 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978544 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978547 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978550 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978553 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978555 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.264531 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978558 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978562 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978565 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978567 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978570 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978573 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978576 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978580 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978584 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978587 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978590 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978593 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978596 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.978599 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980628 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980633 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980636 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980639 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980642 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980645 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.265219 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980647 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980650 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980652 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980655 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980658 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980661 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980663 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980667 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980671 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980673 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980678 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980681 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980684 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980687 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980691 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980694 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980696 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980699 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980702 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.265891 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980708 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980710 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980713 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980716 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980719 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980722 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980725 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980728 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980730 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980733 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980735 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980738 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980740 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980743 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980745 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980748 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980751 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980754 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980756 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980758 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.266487 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980761 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980763 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980766 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980768 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980771 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980773 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980775 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980778 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980781 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980784 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980786 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980789 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.980795 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980921 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980927 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 16:23:16.267146 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980930 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980933 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980935 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980938 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980941 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980943 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980946 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980949 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980951 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980954 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980957 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980959 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980962 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980965 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980969 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980973 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980976 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980979 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980982 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980985 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 16:23:16.267617 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980988 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980991 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980993 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980996 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.980999 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981002 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981005 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981008 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981010 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981015 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981017 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981020 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981022 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981025 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981027 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981030 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981032 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981035 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981037 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981040 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 16:23:16.268268 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981042 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981045 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981047 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981049 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981052 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981054 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981058 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981061 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981063 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981065 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981068 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981070 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981073 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981076 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981078 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981081 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981083 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981086 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981088 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 16:23:16.268877 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981095 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981097 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981101 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981104 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981106 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981109 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981113 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981116 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981119 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981121 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981124 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981127 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981129 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981132 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981134 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981136 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981139 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981142 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981146 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 16:23:16.269436 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981150 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981153 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981156 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981158 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981160 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981163 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981166 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981168 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981171 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981173 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981176 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981178 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981181 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981183 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981186 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981190 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981192 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981195 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 16:23:16.269990 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:15.981197 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.981202 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.981307 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.984246 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.986080 2569 server.go:1019] "Starting client certificate rotation" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.986181 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.986216 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.988564 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:15.989677 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.001618 2569 log.go:25] "Validated CRI v1 runtime API" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.003703 2569 log.go:25] "Validated CRI v1 image API" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.040384 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 16:23:16.270514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.043459 2569 fs.go:135] Filesystem UUIDs: map[4e876c6b-1408-4c9c-b290-46273388be49:/dev/nvme0n1p4 51f51db8-d1fc-493e-a689-d5b10287154a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 07 16:23:16.270881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.043473 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 16:23:16.270881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.047575 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.048074 2569 manager.go:216] Machine: {Timestamp:2026-07-07 16:23:16.046891133 +0000 UTC m=+0.355730404 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118035 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d47e0a2b0447c09da07f16072a520 SystemUUID:ec2d47e0-a2b0-447c-09da-07f16072a520 BootID:fbfb915f-347a-4804-97c1-246d3bc6e21c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:92:21:e9:ba:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:92:21:e9:ba:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:30:29:ab:e1:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.048173 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.048256 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.050975 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.050998 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.051725 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.051736 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.051744 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.052508 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.054133 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.056801 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.056961 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.056991 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057004 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057029 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057042 2569 kubelet.go:398] "Adding apiserver pod source" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057054 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 16:23:16.271061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057326 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.057340 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.061030 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.062943 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.062970 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064077 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064090 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064096 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064100 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064109 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064113 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064121 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064125 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064132 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064137 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064145 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064151 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064793 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.064799 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.067555 2569 server.go:1268] "Started kubelet" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.067658 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.067635 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.067864 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.073946 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.074423 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.074599 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.074689 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.074716 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wmgm6" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.074809 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.075438 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.076883 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wmgm6" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.076861 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-74.ec2.internal.18c00e802a54f8dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-74.ec2.internal,UID:ip-10-0-142-74.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-74.ec2.internal,},FirstTimestamp:2026-07-07 16:23:16.067530973 +0000 UTC m=+0.376370242,LastTimestamp:2026-07-07 16:23:16.067530973 +0000 UTC m=+0.376370242,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-74.ec2.internal,}" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.079407 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.079421 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.079593 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080119 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080124 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080140 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080248 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080254 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 07 16:23:16.271645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080336 2569 factory.go:153] Registering CRI-O factory Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080359 2569 factory.go:223] Registration of the crio container factory successfully Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080403 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080412 2569 factory.go:55] Registering systemd factory Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080419 2569 factory.go:223] Registration of the systemd container factory successfully Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080438 2569 factory.go:103] Registering Raw factory Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080451 2569 manager.go:1196] Started watching for new ooms in manager Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.080823 2569 manager.go:349] Starting recovery of all containers Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.081907 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.082015 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.090494 2569 manager.go:354] Recovery completed Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.091607 2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.094388 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096227 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096256 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096270 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096787 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096795 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096808 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.096872 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097676 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097712 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097724 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097921 2569 policy_none.go:49] "None policy: Start" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097933 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.097942 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.098962 2569 policy_none.go:47] "Start" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142415 2569 manager.go:337] "Starting Device Plugin manager" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.142538 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142551 2569 server.go:87] "Starting device plugin registration server" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142752 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142762 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142861 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142977 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.142987 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.143375 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.143419 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.243266 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.247371 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.247403 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.247413 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.247433 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.257337 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.257367 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-74.ec2.internal\": node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.259482 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 16:23:16.273101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.260857 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 16:23:16.274542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.260893 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 16:23:16.274542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.260915 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 16:23:16.274542 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.260950 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 16:23:16.274542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.264380 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.291440 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.361279 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal"] Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.361325 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.362054 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.362078 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.362087 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.363378 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.363487 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.363511 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364043 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364060 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364082 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364091 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364066 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.364151 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.365198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.365218 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.365823 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientMemory" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.365876 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.365890 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeHasSufficientPID" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.379419 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-74.ec2.internal\" not found" node="ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.382016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.382037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.382056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2dca9c09f5a6c3fe6fd4a6f6555384fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-74.ec2.internal\" (UID: \"2dca9c09f5a6c3fe6fd4a6f6555384fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.383210 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-74.ec2.internal\" not found" node="ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.391722 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.550249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2dca9c09f5a6c3fe6fd4a6f6555384fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-74.ec2.internal\" (UID: \"2dca9c09f5a6c3fe6fd4a6f6555384fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.551681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2dca9c09f5a6c3fe6fd4a6f6555384fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-74.ec2.internal\" (UID: \"2dca9c09f5a6c3fe6fd4a6f6555384fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.551681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482505 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.551681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.482513 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/377b866923178cddcb130a6f4bfa6189-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal\" (UID: \"377b866923178cddcb130a6f4bfa6189\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.551681 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.492351 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.841912 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.592444 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.841912 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.686863 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.841912 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.686876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:16.841912 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.692671 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:16.841912 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.793183 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.894080 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.984895 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:16.985004 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:16.994947 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.070291 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.081249 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 16:18:16 +0000 UTC" deadline="2028-03-22 03:33:30.034433794 +0000 UTC" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.081269 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14963h10m12.953167743s" Jul 07 16:23:17.120322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.095117 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.181591 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.196114 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.275904 2569 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator_377b866923178cddcb130a6f4bfa6189_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.275968 2569 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator_377b866923178cddcb130a6f4bfa6189_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.275991 2569 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = creating pod sandbox with name \"k8s_kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator_377b866923178cddcb130a6f4bfa6189_0\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.276041 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator(377b866923178cddcb130a6f4bfa6189)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator(377b866923178cddcb130a6f4bfa6189)\\\": rpc error: code = Unknown desc = creating pod sandbox with name \\\"k8s_kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal_openshift-machine-config-operator_377b866923178cddcb130a6f4bfa6189_0\\\": unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" podUID="377b866923178cddcb130a6f4bfa6189" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.296366 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:17.353268 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dca9c09f5a6c3fe6fd4a6f6555384fa.slice/crio-c06b3cc45f73adcb3891ffec5f7cce8328fdc5d82f41e98fd03184ae547e32ac WatchSource:0}: Error finding container c06b3cc45f73adcb3891ffec5f7cce8328fdc5d82f41e98fd03184ae547e32ac: Status 404 returned error can't find the container with id c06b3cc45f73adcb3891ffec5f7cce8328fdc5d82f41e98fd03184ae547e32ac Jul 07 16:23:17.381432 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.357596 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:23:17.530182 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.396521 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.530182 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.496691 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.597842 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:17.596988 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-74.ec2.internal\" not found" Jul 07 16:23:17.844434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.613752 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 16:23:17.844434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.681967 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:17.844434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.697934 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 16:23:17.844434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.698944 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" Jul 07 16:23:17.844434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:17.709045 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.033085 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.058074 2569 apiserver.go:52] "Watching apiserver" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.065900 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.066279 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ghsxc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv","openshift-cluster-node-tuning-operator/tuned-lf8vk","openshift-image-registry/node-ca-wnbfm","openshift-multus/multus-9pxgn","openshift-network-diagnostics/network-check-target-7gkjz","openshift-network-operator/iptables-alerter-vtzzm","openshift-ovn-kubernetes/ovnkube-node-wm9gg","kube-system/global-pull-secret-syncer-nk2sf","kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal","openshift-dns/node-resolver-q5fq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal","openshift-multus/multus-additional-cni-plugins-kwdg7","openshift-multus/network-metrics-daemon-cvcbn"] Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.068052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.069361 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.070681 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.070845 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.070874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lbgbr\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.070968 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.071076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.071141 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.071191 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.071746 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.072058 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.072536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.072907 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.072999 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.073261 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.073346 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qdj57\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.074095 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.074590 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.074694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.075067 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.075101 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gvp8m\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.075265 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.077455 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.077682 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.077895 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb54p\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.078002 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.078159 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.079538 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.080202 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 16:23:18.094733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.080501 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7bmlt\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.080570 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.080658 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.081229 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.092899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.093207 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.093676 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wf9lr\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.093678 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.093838 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.094960 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rxp2x\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.094985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2848dd39-1552-4818-8db6-ee928993d131-host-slash\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-modprobe-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysconfig\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-conf\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-run\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095125 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-lib-modules\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095152 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-systemd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-log-socket\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-socket-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095303 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 16:23:18.097097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095306 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-registration-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095371 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-cnibin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-multus-certs\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095499 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-netd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-multus\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.095749 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096323 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-ovn\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovn-node-metrics-cert\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096382 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-host\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfbf\" (UniqueName: \"kubernetes.io/projected/2848dd39-1552-4818-8db6-ee928993d131-kube-api-access-5kfbf\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-sys\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-tmp\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-var-lib-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-netns\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.098409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-etc-kubernetes\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096604 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-system-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096621 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-os-release\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096643 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-hostroot\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-device-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxrd\" (UniqueName: \"kubernetes.io/projected/e111e1f7-d6d8-4885-8498-e33f77335887-kube-api-access-5dxrd\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-cni-binary-copy\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-conf-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-bin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-kubelet\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-multus-daemon-config\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-serviceca\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.096994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-kubernetes\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-var-lib-kubelet\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-slash\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.099413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097068 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-netns\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-socket-dir-parent\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-k8s-cni-cncf-io\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-systemd-units\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-bin\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097227 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-env-overrides\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-script-lib\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-node-log\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-agent-certs\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-sys-fs\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2848dd39-1552-4818-8db6-ee928993d131-iptables-alerter-script\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkz4\" (UniqueName: \"kubernetes.io/projected/5782afaf-d97c-4932-a8bf-f724b3359b57-kube-api-access-6bkz4\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097496 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-host\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-kubelet\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097572 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-etc-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.100253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-konnectivity-ca\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-systemd\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.097988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtzb\" (UniqueName: \"kubernetes.io/projected/05e4f0ff-74c0-4ff2-a230-e111bd53b579-kube-api-access-qqtzb\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-tuned\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/139b6b3e-d026-44b3-9983-fd175f5874d8-kube-api-access-wcdbq\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-config\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf7q\" (UniqueName: \"kubernetes.io/projected/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-kube-api-access-wgf7q\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098205 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098272 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5lwz9\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098478 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.098795 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.098876 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.099864 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.099968 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-745zh\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.100848 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 16:23:18.102538 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.101026 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.120276 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcnrv" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.130059 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcnrv" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.181642 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxrd\" (UniqueName: \"kubernetes.io/projected/e111e1f7-d6d8-4885-8498-e33f77335887-kube-api-access-5dxrd\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9a61f94-1bf2-4834-b9d3-e90c846494c1-hosts-file\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-cni-binary-copy\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-conf-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.198984 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-conf-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-bin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-kubelet\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-bin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-multus-daemon-config\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.254656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-serviceca\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-kubelet\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-kubernetes\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-var-lib-kubelet\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qp47\" (UniqueName: \"kubernetes.io/projected/cfb4efee-7e9b-42dd-b216-07f03d7365e9-kube-api-access-8qp47\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-slash\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-netns\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199522 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-kubernetes\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-socket-dir-parent\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199560 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-k8s-cni-cncf-io\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-systemd-units\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-var-lib-kubelet\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-bin\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-cni-binary-copy\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-env-overrides\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-script-lib\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-node-log\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199703 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-socket-dir-parent\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199722 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-slash\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199746 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-netns\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-agent-certs\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-sys-fs\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-systemd-units\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199814 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-serviceca\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2848dd39-1552-4818-8db6-ee928993d131-iptables-alerter-script\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkz4\" (UniqueName: \"kubernetes.io/projected/5782afaf-d97c-4932-a8bf-f724b3359b57-kube-api-access-6bkz4\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-host\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-bin\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199926 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e111e1f7-d6d8-4885-8498-e33f77335887-multus-daemon-config\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a61f94-1bf2-4834-b9d3-e90c846494c1-tmp-dir\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199829 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-k8s-cni-cncf-io\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.199995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-kubelet\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-host\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-etc-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.256762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200052 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-node-log\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-konnectivity-ca\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-system-cni-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200159 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-sys-fs\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-os-release\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-kubelet-config\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-systemd\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200247 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-env-overrides\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtzb\" (UniqueName: \"kubernetes.io/projected/05e4f0ff-74c0-4ff2-a230-e111bd53b579-kube-api-access-qqtzb\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-etc-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-run-ovn-kubernetes\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-tuned\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/139b6b3e-d026-44b3-9983-fd175f5874d8-kube-api-access-wcdbq\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200540 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.259403 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200553 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2848dd39-1552-4818-8db6-ee928993d131-iptables-alerter-script\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-script-lib\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200586 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200607 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-systemd\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-config\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200803 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-kubelet\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf7q\" (UniqueName: \"kubernetes.io/projected/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-kube-api-access-wgf7q\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2848dd39-1552-4818-8db6-ee928993d131-host-slash\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-modprobe-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysconfig\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.200996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rx2\" (UniqueName: \"kubernetes.io/projected/c9a61f94-1bf2-4834-b9d3-e90c846494c1-kube-api-access-l4rx2\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-conf\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-run\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-lib-modules\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5bj\" (UniqueName: \"kubernetes.io/projected/0d189f61-09ee-44d8-a2e0-38b22a416ad4-kube-api-access-qb5bj\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-systemd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.260223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-log-socket\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201224 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovnkube-config\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201258 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-systemd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2848dd39-1552-4818-8db6-ee928993d131-host-slash\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201342 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-konnectivity-ca\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201365 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-lib-modules\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-conf\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201420 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysconfig\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-socket-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-registration-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-modprobe-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201500 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-socket-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-log-socket\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-cnibin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201541 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-registration-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.261025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-multus-certs\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201570 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-run\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201606 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-cnibin\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-netd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201617 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-multus-certs\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201645 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cnibin\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-multus\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-host-cni-netd\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201727 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-dbus\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-var-lib-cni-multus\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201764 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-sysctl-d\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-ovn\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovn-node-metrics-cert\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201876 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-run-ovn\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-host\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.261610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201915 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-host\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfbf\" (UniqueName: \"kubernetes.io/projected/2848dd39-1552-4818-8db6-ee928993d131-kube-api-access-5kfbf\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-sys\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.201974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-tmp\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/139b6b3e-d026-44b3-9983-fd175f5874d8-sys\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-var-lib-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-netns\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-etc-kubernetes\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05e4f0ff-74c0-4ff2-a230-e111bd53b579-var-lib-openvswitch\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-system-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-host-run-netns\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-os-release\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-hostroot\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202218 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-etc-kubernetes\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202251 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-multus-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202272 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-hostroot\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202279 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-os-release\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.262490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e111e1f7-d6d8-4885-8498-e33f77335887-system-cni-dir\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-device-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.202381 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5782afaf-d97c-4932-a8bf-f724b3359b57-device-dir\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.205019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-tmp\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.205403 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/139b6b3e-d026-44b3-9983-fd175f5874d8-etc-tuned\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.205837 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05e4f0ff-74c0-4ff2-a230-e111bd53b579-ovn-node-metrics-cert\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.206267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b-agent-certs\") pod \"konnectivity-agent-ghsxc\" (UID: \"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b\") " pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.211216 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.211264 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.211278 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.211345 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:18.711317482 +0000 UTC m=+3.020156738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.214169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdbq\" (UniqueName: \"kubernetes.io/projected/139b6b3e-d026-44b3-9983-fd175f5874d8-kube-api-access-wcdbq\") pod \"tuned-lf8vk\" (UID: \"139b6b3e-d026-44b3-9983-fd175f5874d8\") " pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.214764 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfbf\" (UniqueName: \"kubernetes.io/projected/2848dd39-1552-4818-8db6-ee928993d131-kube-api-access-5kfbf\") pod \"iptables-alerter-vtzzm\" (UID: \"2848dd39-1552-4818-8db6-ee928993d131\") " pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.215360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkz4\" (UniqueName: \"kubernetes.io/projected/5782afaf-d97c-4932-a8bf-f724b3359b57-kube-api-access-6bkz4\") pod \"aws-ebs-csi-driver-node-2r2rv\" (UID: \"5782afaf-d97c-4932-a8bf-f724b3359b57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.215355 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtzb\" (UniqueName: \"kubernetes.io/projected/05e4f0ff-74c0-4ff2-a230-e111bd53b579-kube-api-access-qqtzb\") pod \"ovnkube-node-wm9gg\" (UID: \"05e4f0ff-74c0-4ff2-a230-e111bd53b579\") " pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.215559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf7q\" (UniqueName: \"kubernetes.io/projected/0e9f2128-48f5-4b83-816b-eba10e7a1bf2-kube-api-access-wgf7q\") pod \"node-ca-wnbfm\" (UID: \"0e9f2128-48f5-4b83-816b-eba10e7a1bf2\") " pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.263114 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.216252 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxrd\" (UniqueName: \"kubernetes.io/projected/e111e1f7-d6d8-4885-8498-e33f77335887-kube-api-access-5dxrd\") pod \"multus-9pxgn\" (UID: \"e111e1f7-d6d8-4885-8498-e33f77335887\") " pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.267731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" event={"ID":"2dca9c09f5a6c3fe6fd4a6f6555384fa","Type":"ContainerStarted","Data":"c06b3cc45f73adcb3891ffec5f7cce8328fdc5d82f41e98fd03184ae547e32ac"} Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.267774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.267976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.276782 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377b866923178cddcb130a6f4bfa6189.slice/crio-315882fbb4585b1db7667c2e847dc75e32b00dcdb57413989548585b11940e2a WatchSource:0}: Error finding container 315882fbb4585b1db7667c2e847dc75e32b00dcdb57413989548585b11940e2a: Status 404 returned error can't find the container with id 315882fbb4585b1db7667c2e847dc75e32b00dcdb57413989548585b11940e2a Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303456 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a61f94-1bf2-4834-b9d3-e90c846494c1-tmp-dir\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-system-cni-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-os-release\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-kubelet-config\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303583 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-system-cni-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303652 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-os-release\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303660 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rx2\" (UniqueName: \"kubernetes.io/projected/c9a61f94-1bf2-4834-b9d3-e90c846494c1-kube-api-access-l4rx2\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303769 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9a61f94-1bf2-4834-b9d3-e90c846494c1-tmp-dir\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303797 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-kubelet-config\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303809 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.381347 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.303851 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5bj\" (UniqueName: \"kubernetes.io/projected/0d189f61-09ee-44d8-a2e0-38b22a416ad4-kube-api-access-qb5bj\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cnibin\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.303944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cnibin\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.303964 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:18.803943907 +0000 UTC m=+3.112783178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-dbus\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9a61f94-1bf2-4834-b9d3-e90c846494c1-hosts-file\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9a61f94-1bf2-4834-b9d3-e90c846494c1-hosts-file\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304224 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f6620e90-14c4-4008-a09e-d56578336e2a-dbus\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qp47\" (UniqueName: \"kubernetes.io/projected/cfb4efee-7e9b-42dd-b216-07f03d7365e9-kube-api-access-8qp47\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304272 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.304321 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.304377 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:18.804360837 +0000 UTC m=+3.113200109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:18.382651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304524 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.383509 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.304686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfb4efee-7e9b-42dd-b216-07f03d7365e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.383509 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.316945 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rx2\" (UniqueName: \"kubernetes.io/projected/c9a61f94-1bf2-4834-b9d3-e90c846494c1-kube-api-access-l4rx2\") pod \"node-resolver-q5fq8\" (UID: \"c9a61f94-1bf2-4834-b9d3-e90c846494c1\") " pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.383509 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.318540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qp47\" (UniqueName: \"kubernetes.io/projected/cfb4efee-7e9b-42dd-b216-07f03d7365e9-kube-api-access-8qp47\") pod \"multus-additional-cni-plugins-kwdg7\" (UID: \"cfb4efee-7e9b-42dd-b216-07f03d7365e9\") " pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.383509 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.319744 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5bj\" (UniqueName: \"kubernetes.io/projected/0d189f61-09ee-44d8-a2e0-38b22a416ad4-kube-api-access-qb5bj\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.383509 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.380192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9pxgn" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.388435 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode111e1f7_d6d8_4885_8498_e33f77335887.slice/crio-52b33f3457c1f9fb22bb4bead127424e7d0d289c5315be9217ada35c9f36c24d WatchSource:0}: Error finding container 52b33f3457c1f9fb22bb4bead127424e7d0d289c5315be9217ada35c9f36c24d: Status 404 returned error can't find the container with id 52b33f3457c1f9fb22bb4bead127424e7d0d289c5315be9217ada35c9f36c24d Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.393792 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.403954 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e4f0ff_74c0_4ff2_a230_e111bd53b579.slice/crio-9bbe29afba1a95e8ebfef4bf2372a5bf13c7ab2204f3e4cfa97ab46ef6bf357f WatchSource:0}: Error finding container 9bbe29afba1a95e8ebfef4bf2372a5bf13c7ab2204f3e4cfa97ab46ef6bf357f: Status 404 returned error can't find the container with id 9bbe29afba1a95e8ebfef4bf2372a5bf13c7ab2204f3e4cfa97ab46ef6bf357f Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.426886 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnbfm" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.433361 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vtzzm" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.433622 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9f2128_48f5_4b83_816b_eba10e7a1bf2.slice/crio-f4212eee382fab364afb217f13b942028dc6bd238c64c78d4ec8f4e270b75c5a WatchSource:0}: Error finding container f4212eee382fab364afb217f13b942028dc6bd238c64c78d4ec8f4e270b75c5a: Status 404 returned error can't find the container with id f4212eee382fab364afb217f13b942028dc6bd238c64c78d4ec8f4e270b75c5a Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.439938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.440154 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2848dd39_1552_4818_8db6_ee928993d131.slice/crio-a88ffc9aceaac96df977a7c077694ade552fa66e9e68ba39427d5d4b267f7846 WatchSource:0}: Error finding container a88ffc9aceaac96df977a7c077694ade552fa66e9e68ba39427d5d4b267f7846: Status 404 returned error can't find the container with id a88ffc9aceaac96df977a7c077694ade552fa66e9e68ba39427d5d4b267f7846 Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.448640 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.454359 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139b6b3e_d026_44b3_9983_fd175f5874d8.slice/crio-c249e922ca4017511a83dad3f7e54699446b3f080bb8d5d4dd49883b020b210c WatchSource:0}: Error finding container c249e922ca4017511a83dad3f7e54699446b3f080bb8d5d4dd49883b020b210c: Status 404 returned error can't find the container with id c249e922ca4017511a83dad3f7e54699446b3f080bb8d5d4dd49883b020b210c Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.458143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.461211 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34608c24_a6e1_4a49_9cc7_2f5bd7fe8b6b.slice/crio-228a7214b025a80403042d54b0edbf486cdfb2eeeedfe73418c806aeb328e253 WatchSource:0}: Error finding container 228a7214b025a80403042d54b0edbf486cdfb2eeeedfe73418c806aeb328e253: Status 404 returned error can't find the container with id 228a7214b025a80403042d54b0edbf486cdfb2eeeedfe73418c806aeb328e253 Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.466559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5fq8" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.473788 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5782afaf_d97c_4932_a8bf_f724b3359b57.slice/crio-3de2dc262d83a31d58089fcd24afea57d334c2d5f41aecbb7bb32a933255c25a WatchSource:0}: Error finding container 3de2dc262d83a31d58089fcd24afea57d334c2d5f41aecbb7bb32a933255c25a: Status 404 returned error can't find the container with id 3de2dc262d83a31d58089fcd24afea57d334c2d5f41aecbb7bb32a933255c25a Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.474873 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.477096 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a61f94_1bf2_4834_b9d3_e90c846494c1.slice/crio-223f36f12ccf45493cd3343eb71a47f90940e0da3708fa095c3c458d8717885b WatchSource:0}: Error finding container 223f36f12ccf45493cd3343eb71a47f90940e0da3708fa095c3c458d8717885b: Status 404 returned error can't find the container with id 223f36f12ccf45493cd3343eb71a47f90940e0da3708fa095c3c458d8717885b Jul 07 16:23:18.518486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:18.484054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb4efee_7e9b_42dd_b216_07f03d7365e9.slice/crio-b2bb511ca786bfa5c8b97ea98aaa108fcfa7f857128c2e77c3cb79cde5f00b8e WatchSource:0}: Error finding container b2bb511ca786bfa5c8b97ea98aaa108fcfa7f857128c2e77c3cb79cde5f00b8e: Status 404 returned error can't find the container with id b2bb511ca786bfa5c8b97ea98aaa108fcfa7f857128c2e77c3cb79cde5f00b8e Jul 07 16:23:18.808500 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.808413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:18.808668 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.808498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:18.808668 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:18.808569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:18.808787 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.808740 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:18.808838 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.808819 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:19.80879678 +0000 UTC m=+4.117636063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:18.809435 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809407 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:18.809435 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809435 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:18.809633 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809458 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:18.809633 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809512 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:19.809492615 +0000 UTC m=+4.118331889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:18.809633 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809607 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:18.809792 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:18.809656 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:19.809640157 +0000 UTC m=+4.118479420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:19.132660 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.132544 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:18:18 +0000 UTC" deadline="2028-01-14 06:46:35.211144824 +0000 UTC" Jul 07 16:23:19.132660 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.132605 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13334h23m16.078544327s" Jul 07 16:23:19.271661 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.271613 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5fq8" event={"ID":"c9a61f94-1bf2-4834-b9d3-e90c846494c1","Type":"ContainerStarted","Data":"223f36f12ccf45493cd3343eb71a47f90940e0da3708fa095c3c458d8717885b"} Jul 07 16:23:19.273493 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.273426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" event={"ID":"5782afaf-d97c-4932-a8bf-f724b3359b57","Type":"ContainerStarted","Data":"3de2dc262d83a31d58089fcd24afea57d334c2d5f41aecbb7bb32a933255c25a"} Jul 07 16:23:19.277279 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.277248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghsxc" event={"ID":"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b","Type":"ContainerStarted","Data":"228a7214b025a80403042d54b0edbf486cdfb2eeeedfe73418c806aeb328e253"} Jul 07 16:23:19.278802 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.278776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" event={"ID":"139b6b3e-d026-44b3-9983-fd175f5874d8","Type":"ContainerStarted","Data":"c249e922ca4017511a83dad3f7e54699446b3f080bb8d5d4dd49883b020b210c"} Jul 07 16:23:19.280551 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.280523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vtzzm" event={"ID":"2848dd39-1552-4818-8db6-ee928993d131","Type":"ContainerStarted","Data":"a88ffc9aceaac96df977a7c077694ade552fa66e9e68ba39427d5d4b267f7846"} Jul 07 16:23:19.282257 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.282233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnbfm" event={"ID":"0e9f2128-48f5-4b83-816b-eba10e7a1bf2","Type":"ContainerStarted","Data":"f4212eee382fab364afb217f13b942028dc6bd238c64c78d4ec8f4e270b75c5a"} Jul 07 16:23:19.287174 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.287119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9pxgn" event={"ID":"e111e1f7-d6d8-4885-8498-e33f77335887","Type":"ContainerStarted","Data":"52b33f3457c1f9fb22bb4bead127424e7d0d289c5315be9217ada35c9f36c24d"} Jul 07 16:23:19.288991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.288962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerStarted","Data":"b2bb511ca786bfa5c8b97ea98aaa108fcfa7f857128c2e77c3cb79cde5f00b8e"} Jul 07 16:23:19.290752 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.290725 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"9bbe29afba1a95e8ebfef4bf2372a5bf13c7ab2204f3e4cfa97ab46ef6bf357f"} Jul 07 16:23:19.292840 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.292813 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" event={"ID":"377b866923178cddcb130a6f4bfa6189","Type":"ContainerStarted","Data":"315882fbb4585b1db7667c2e847dc75e32b00dcdb57413989548585b11940e2a"} Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.815225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.815286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:19.815327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.815482 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.815546 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:21.815525447 +0000 UTC m=+6.124364713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.815975 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.816032 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:21.816014195 +0000 UTC m=+6.124853454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.816117 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.816137 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.816150 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:19.816279 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:19.816189 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:21.81617618 +0000 UTC m=+6.125015453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:20.133132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.132995 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 16:18:18 +0000 UTC" deadline="2028-01-21 15:46:09.716291803 +0000 UTC" Jul 07 16:23:20.133132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.133035 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13511h22m49.58326198s" Jul 07 16:23:20.263082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.263054 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:20.263252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.263056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:20.263252 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:20.263185 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:20.263373 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:20.263304 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:20.263373 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.263056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:20.263475 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:20.263431 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:20.320827 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.320791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" event={"ID":"2dca9c09f5a6c3fe6fd4a6f6555384fa","Type":"ContainerStarted","Data":"eb258c2ea8b21d7727835605789a75fa09c8e6360d539a16aec492269908026e"} Jul 07 16:23:20.342624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:20.342565 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-74.ec2.internal" podStartSLOduration=2.3425471030000002 podStartE2EDuration="2.342547103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:20.341808752 +0000 UTC m=+4.650648081" watchObservedRunningTime="2026-07-07 16:23:20.342547103 +0000 UTC m=+4.651386387" Jul 07 16:23:21.834167 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:21.833915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:21.834641 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:21.834210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:21.834641 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:21.834257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:21.834641 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.834393 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:21.834641 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.834457 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.834436905 +0000 UTC m=+10.143276181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:21.835092 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.834852 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:21.835092 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.834907 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.83488836 +0000 UTC m=+10.143727622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:21.835092 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.835038 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:21.835092 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.835061 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:21.835092 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.835074 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:21.835371 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:21.835117 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:25.835102354 +0000 UTC m=+10.143941612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:22.261642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:22.261790 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:22.261922 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:22.262052 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:22.262114 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:22.264192 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:22.262185 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:24.261352 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:24.261495 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:24.261929 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:24.262037 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:24.262137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:24.262312 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:24.262209 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:25.869086 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:25.869044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:25.869523 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:25.869104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:25.869523 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:25.869136 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:25.869523 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869247 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:25.869523 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869309 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:33.86928931 +0000 UTC m=+18.178128577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:25.869767 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869677 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:25.869767 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869728 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:33.869712887 +0000 UTC m=+18.178552145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:25.869903 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869827 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:25.869903 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869869 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:25.869903 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869882 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:25.870035 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:25.869935 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:33.869918561 +0000 UTC m=+18.178757831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:26.262798 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:26.262720 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:26.262975 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:26.262850 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:26.263224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:26.263201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:26.263334 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:26.263316 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:26.263394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:26.263363 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:26.263448 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:26.263431 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:28.261788 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:28.261755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:28.262282 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:28.261914 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:28.262449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:28.262305 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:28.262449 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:28.262400 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:28.263848 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:28.263689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:28.263848 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:28.263798 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:30.261481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:30.261402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:30.261481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:30.261423 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:30.261972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:30.261402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:30.261972 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:30.261523 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:30.261972 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:30.261603 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:30.261972 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:30.261743 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:32.261202 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:32.261156 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:32.261689 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:32.261166 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:32.261689 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:32.261285 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:32.261689 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:32.261311 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:32.261689 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:32.261397 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:32.261689 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:32.261495 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:33.928053 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:33.928018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:33.928063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:33.928101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928177 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928201 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928251 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:23:49.928237046 +0000 UTC m=+34.237076307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928206 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928287 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928329 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:49.92831835 +0000 UTC m=+34.237157606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928183 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:33.928487 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:33.928365 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:49.928355127 +0000 UTC m=+34.237194388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:34.262208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:34.262127 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:34.262208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:34.262143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:34.262392 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:34.262261 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:34.262392 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:34.262340 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:34.262392 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:34.262358 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:34.262511 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:34.262442 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:36.262534 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:36.262500 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:36.263001 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:36.262615 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:36.263001 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:36.262623 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:36.263001 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:36.262650 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:36.263001 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:36.262707 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:36.263001 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:36.262774 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:38.261477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.261332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:38.262171 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.261340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:38.262171 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:38.261576 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:38.262171 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.261370 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:38.262171 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:38.261684 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:38.262171 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:38.261767 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:38.361262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.361233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"a02762bed575d191f8900c651670389570f16a7047021a9e5abe1a796be076a5"} Jul 07 16:23:38.361371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.361323 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"3e23b17a37d66b8d403119b31ed94bb2cfbcad26523789fa2228d185cf7b5811"} Jul 07 16:23:38.361432 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.361374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"490bcb8a97d9049887a69a0d21cf6d363b1a72e8e62dfe10b0878e23c1294097"} Jul 07 16:23:38.362705 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.362679 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" event={"ID":"139b6b3e-d026-44b3-9983-fd175f5874d8","Type":"ContainerStarted","Data":"db22c0a5552ac18ce59247e5bb54e1ffe4f5af71b4cf6c4132d3357d464b81fa"} Jul 07 16:23:38.364318 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.364296 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9pxgn" event={"ID":"e111e1f7-d6d8-4885-8498-e33f77335887","Type":"ContainerStarted","Data":"b237d3ce91b7cbaaf69460e7b7f4bccbbcba642e304894c213016a803eb27165"} Jul 07 16:23:38.383612 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.383549 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lf8vk" podStartSLOduration=1.832341219 podStartE2EDuration="21.383537042s" totalImagesPullingTime="19.551195823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:38.383086938 +0000 UTC m=+22.691926217" watchObservedRunningTime="2026-07-07 16:23:38.383537042 +0000 UTC m=+22.692376319" Jul 07 16:23:38.404072 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:38.404032 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9pxgn" podStartSLOduration=2.719484389 podStartE2EDuration="22.40402027s" totalImagesPullingTime="19.684535881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:38.403776912 +0000 UTC m=+22.712616203" watchObservedRunningTime="2026-07-07 16:23:38.40402027 +0000 UTC m=+22.712859549" Jul 07 16:23:39.368669 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.368454 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnbfm" event={"ID":"0e9f2128-48f5-4b83-816b-eba10e7a1bf2","Type":"ContainerStarted","Data":"94dea54f70838494476f0800ddc3ee322b338f98affa8a561c46c73013f25075"} Jul 07 16:23:39.370148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.370124 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="3f7226190b2f444bfb952df6c9e1acd43126326b5a9e56f6ec9b756cd1b136a6" exitCode=0 Jul 07 16:23:39.370288 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.370162 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"3f7226190b2f444bfb952df6c9e1acd43126326b5a9e56f6ec9b756cd1b136a6"} Jul 07 16:23:39.374024 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.373998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"daaaca157b09eddeff2748929c8cc62aaf14d5787ca571964519aa99dfcdc81f"} Jul 07 16:23:39.374123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.374063 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"978511df28d84cb5204a7d257b7c2ddc264f727f8a2a723d645adc47f2722819"} Jul 07 16:23:39.374123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.374112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"bb4c857236ef883b01f70bca49cb8623e6090ae8f143fb0884643e66043d03d5"} Jul 07 16:23:39.375702 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.375665 2569 generic.go:356] "Generic (PLEG): container finished" podID="377b866923178cddcb130a6f4bfa6189" containerID="bed64ae557bbf442d9b76d2c61bf6d258e7253ac16ab3fa65b7dbce32adf663e" exitCode=0 Jul 07 16:23:39.375815 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.375728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" event={"ID":"377b866923178cddcb130a6f4bfa6189","Type":"ContainerDied","Data":"bed64ae557bbf442d9b76d2c61bf6d258e7253ac16ab3fa65b7dbce32adf663e"} Jul 07 16:23:39.377455 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.377436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5fq8" event={"ID":"c9a61f94-1bf2-4834-b9d3-e90c846494c1","Type":"ContainerStarted","Data":"1cff897a873bdec14d204fcc2c410ce4186afec0c8a57eb0a0653f0e8e400765"} Jul 07 16:23:39.378989 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.378967 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" event={"ID":"5782afaf-d97c-4932-a8bf-f724b3359b57","Type":"ContainerStarted","Data":"cd8939e9a7841ac5d768eb7c02fd8a79463f8cf7ad59e49fd82514d841b5c8b5"} Jul 07 16:23:39.380540 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.380513 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghsxc" event={"ID":"34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b","Type":"ContainerStarted","Data":"29ad8b75fd2584deb028a2d193706b97105a50626f861bf0a00f314388d07ac4"} Jul 07 16:23:39.387822 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.387786 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wnbfm" podStartSLOduration=3.834849193 podStartE2EDuration="23.387771935s" totalImagesPullingTime="19.552922742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:39.387361844 +0000 UTC m=+23.696201126" watchObservedRunningTime="2026-07-07 16:23:39.387771935 +0000 UTC m=+23.696611219" Jul 07 16:23:39.405565 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.405524 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q5fq8" podStartSLOduration=6.90271536 podStartE2EDuration="22.405507139s" totalImagesPullingTime="15.502791779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:39.405395774 +0000 UTC m=+23.714235053" watchObservedRunningTime="2026-07-07 16:23:39.405507139 +0000 UTC m=+23.714346418" Jul 07 16:23:39.441774 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.441735 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ghsxc" podStartSLOduration=2.923527375 podStartE2EDuration="22.441722541s" totalImagesPullingTime="19.518195166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:39.441530203 +0000 UTC m=+23.750369481" watchObservedRunningTime="2026-07-07 16:23:39.441722541 +0000 UTC m=+23.750561822" Jul 07 16:23:39.946073 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:39.946039 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 16:23:40.157143 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.156980 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T16:23:39.946072094Z","UUID":"028583a4-c393-4257-9cf8-7833d0470d34","Handler":null,"Name":"","Endpoint":""} Jul 07 16:23:40.158519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.158498 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 16:23:40.158649 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.158538 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 16:23:40.261801 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.261776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:40.261981 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.261776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:40.261981 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:40.261903 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:40.261981 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:40.261961 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:40.262201 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.262027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:40.262201 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:40.262110 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:40.384316 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.384282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vtzzm" event={"ID":"2848dd39-1552-4818-8db6-ee928993d131","Type":"ContainerStarted","Data":"395f43de36b8848dd4b1c243b8ad9fbcda61b7850f90a2768fa04454c0996c02"} Jul 07 16:23:40.386436 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.386411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" event={"ID":"377b866923178cddcb130a6f4bfa6189","Type":"ContainerStarted","Data":"567b46b14763c9914c3e5f6b4b0412ae33cbd3bf45b69a8e503a2dd4cedd1c24"} Jul 07 16:23:40.388252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.388228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" event={"ID":"5782afaf-d97c-4932-a8bf-f724b3359b57","Type":"ContainerStarted","Data":"58bcdc06447ec03ff2207bedb56a1a934e7039dff09fdc21d6425248158a604a"} Jul 07 16:23:40.400768 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:40.400725 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vtzzm" podStartSLOduration=7.8596553799999995 podStartE2EDuration="23.40070925s" totalImagesPullingTime="15.54105387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:40.400616669 +0000 UTC m=+24.709455948" watchObservedRunningTime="2026-07-07 16:23:40.40070925 +0000 UTC m=+24.709548531" Jul 07 16:23:41.392586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:41.392549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" event={"ID":"5782afaf-d97c-4932-a8bf-f724b3359b57","Type":"ContainerStarted","Data":"f8ec8a0844db8d69465f97ae5b196a9100d475a06619c0fc3067eafa72288600"} Jul 07 16:23:41.397370 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:41.397325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"01c9cd1b795da3e588cf83eecfceafad8235b57418c0ee662456cc6c7aae3ee2"} Jul 07 16:23:41.414135 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:41.414089 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r2rv" podStartSLOduration=2.485433397 podStartE2EDuration="24.414077819s" totalImagesPullingTime="21.928644422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:41.413982154 +0000 UTC m=+25.722821433" watchObservedRunningTime="2026-07-07 16:23:41.414077819 +0000 UTC m=+25.722917116" Jul 07 16:23:41.414709 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:41.414680 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-74.ec2.internal" podStartSLOduration=23.41466996 podStartE2EDuration="23.41466996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:40.420181304 +0000 UTC m=+24.729020593" watchObservedRunningTime="2026-07-07 16:23:41.41466996 +0000 UTC m=+25.723509238" Jul 07 16:23:42.261712 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:42.261682 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:42.261897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:42.261735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:42.261975 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:42.261883 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:42.261975 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:42.261935 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:42.261975 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:42.261959 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:42.262129 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:42.262112 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:42.716028 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:42.715997 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:42.716687 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:42.716663 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:43.401557 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:43.401532 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:43.402020 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:43.401996 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghsxc" Jul 07 16:23:44.261716 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.261657 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:44.261716 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.261668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:44.262540 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:44.261746 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:44.262540 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:44.261850 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:44.262540 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.261892 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:44.262540 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:44.261948 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:44.405670 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.405644 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="7b7555de5e44cf661b3bff435cedba40c10ddf1833191fb5a3a7f02da27b85db" exitCode=0 Jul 07 16:23:44.405783 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.405735 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"7b7555de5e44cf661b3bff435cedba40c10ddf1833191fb5a3a7f02da27b85db"} Jul 07 16:23:44.413454 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.413425 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" event={"ID":"05e4f0ff-74c0-4ff2-a230-e111bd53b579","Type":"ContainerStarted","Data":"c4357783029625e72798d1e90513334cc7533950ce37690067c654904880ed88"} Jul 07 16:23:44.498751 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:44.498711 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" podStartSLOduration=8.876910008 podStartE2EDuration="28.498700853s" totalImagesPullingTime="19.621790845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:44.49802647 +0000 UTC m=+28.806865747" watchObservedRunningTime="2026-07-07 16:23:44.498700853 +0000 UTC m=+28.807540166" Jul 07 16:23:45.415619 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.415517 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:45.416123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.415654 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:45.416123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.415682 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:45.430191 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.430169 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:45.430338 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.430273 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:23:45.784537 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.784459 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gkjz"] Jul 07 16:23:45.784650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.784571 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:45.784708 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:45.784670 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:45.786328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.786305 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk2sf"] Jul 07 16:23:45.786433 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.786402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:45.786507 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:45.786488 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:45.789379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.789357 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cvcbn"] Jul 07 16:23:45.789471 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:45.789459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:45.789583 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:45.789564 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:46.418337 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:46.418305 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="0d28da5567a48b166bd6353fe88022426354d361330fa10cea83a9643fefe37c" exitCode=0 Jul 07 16:23:46.418668 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:46.418380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"0d28da5567a48b166bd6353fe88022426354d361330fa10cea83a9643fefe37c"} Jul 07 16:23:47.262085 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:47.262056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:47.262225 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:47.262096 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:47.262225 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:47.262176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:47.262225 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:47.262194 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:47.262343 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:47.262245 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:47.262434 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:47.262401 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:48.424969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:48.424934 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="db6085a056f3663b23ce9cf02c74364f69c25f50eb8b1aaeefda8f7370b75649" exitCode=0 Jul 07 16:23:48.425348 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:48.425000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"db6085a056f3663b23ce9cf02c74364f69c25f50eb8b1aaeefda8f7370b75649"} Jul 07 16:23:49.261785 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.261759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:49.261785 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.261770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:49.262000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.261862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:49.262000 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.261867 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gkjz" podUID="d20d6aed-2688-4b41-bc07-80774e325072" Jul 07 16:23:49.262000 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.261928 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk2sf" podUID="f6620e90-14c4-4008-a09e-d56578336e2a" Jul 07 16:23:49.262122 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.262019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:23:49.947769 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.947738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:49.947769 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.947775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:49.947805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947901 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947938 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947963 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947902 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947978 2569 projected.go:196] Error preparing data for projected volume kube-api-access-j5z9g for pod openshift-network-diagnostics/network-check-target-7gkjz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.947945 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:21.947930861 +0000 UTC m=+66.256770118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.948037 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret podName:f6620e90-14c4-4008-a09e-d56578336e2a nodeName:}" failed. No retries permitted until 2026-07-07 16:24:21.948019849 +0000 UTC m=+66.256859105 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret") pod "global-pull-secret-syncer-nk2sf" (UID: "f6620e90-14c4-4008-a09e-d56578336e2a") : object "kube-system"/"original-pull-secret" not registered Jul 07 16:23:49.948247 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:49.948053 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g podName:d20d6aed-2688-4b41-bc07-80774e325072 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:21.948043539 +0000 UTC m=+66.256882795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j5z9g" (UniqueName: "kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g") pod "network-check-target-7gkjz" (UID: "d20d6aed-2688-4b41-bc07-80774e325072") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 16:23:50.951243 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:50.951219 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-74.ec2.internal" event="NodeReady" Jul 07 16:23:50.951590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:50.951321 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 16:23:50.996574 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:50.996549 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:23:50.998674 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:50.998622 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:50.999924 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:50.999634 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp"] Jul 07 16:23:51.001130 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001105 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 16:23:51.001268 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001131 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 16:23:51.001268 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001250 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47hf8\"" Jul 07 16:23:51.001396 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001271 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh"] Jul 07 16:23:51.001735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001509 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 16:23:51.001735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.001547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" Jul 07 16:23:51.004329 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.003520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.004891 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.004714 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 16:23:51.004891 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.004790 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 16:23:51.005364 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.005346 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hbpgr\"" Jul 07 16:23:51.007977 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.007957 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 16:23:51.008117 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.008068 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 16:23:51.008459 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.008441 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vks4c\"" Jul 07 16:23:51.011380 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.011362 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 16:23:51.013344 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.013323 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp"] Jul 07 16:23:51.015118 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.015075 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:23:51.015788 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.015768 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh"] Jul 07 16:23:51.024105 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.024008 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7htvg"] Jul 07 16:23:51.026028 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.026011 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vljgp"] Jul 07 16:23:51.026146 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.026128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.028018 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.028002 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.028474 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.028456 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 16:23:51.028555 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.028482 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 16:23:51.028555 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.028495 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f9fh\"" Jul 07 16:23:51.030488 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.030464 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 16:23:51.030587 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.030547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f8xb4\"" Jul 07 16:23:51.030644 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.030553 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 16:23:51.030698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.030603 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 16:23:51.040134 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.040117 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vljgp"] Jul 07 16:23:51.040885 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.040854 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7htvg"] Jul 07 16:23:51.154910 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.154884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155030 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.154915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155030 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.154976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwtz\" (UniqueName: \"kubernetes.io/projected/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-kube-api-access-jgwtz\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.155030 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7h4l\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155030 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155020 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f07ac95b-22f0-41cc-8d29-c68702c78185-cert\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxpj\" (UniqueName: \"kubernetes.io/projected/e043eb7e-fee6-4196-b1e7-73c040138f3a-kube-api-access-kjxpj\") pod \"network-check-source-989f8b8dc-6znrp\" (UID: \"e043eb7e-fee6-4196-b1e7-73c040138f3a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-tmp-dir\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.155176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrxh\" (UniqueName: \"kubernetes.io/projected/f07ac95b-22f0-41cc-8d29-c68702c78185-kube-api-access-xhrxh\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-config-volume\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af45c32-632e-4dbb-8f65-9a30f443b58f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155224 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155241 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155256 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.155376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.155269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.255869 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.255869 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-tmp-dir\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrxh\" (UniqueName: \"kubernetes.io/projected/f07ac95b-22f0-41cc-8d29-c68702c78185-kube-api-access-xhrxh\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-config-volume\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af45c32-632e-4dbb-8f65-9a30f443b58f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.255944 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.255958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256062 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256018 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:51.75599754 +0000 UTC m=+36.064836798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:23:51.256378 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-tmp-dir\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.256430 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256386 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.256430 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256513 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256445 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256513 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256485 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:23:51.256583 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256583 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256530 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:51.756515424 +0000 UTC m=+36.065354697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:23:51.256583 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256579 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-config-volume\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256596 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256598 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256610 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwtz\" (UniqueName: \"kubernetes.io/projected/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-kube-api-access-jgwtz\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.256680 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.256659 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:51.756644593 +0000 UTC m=+36.065483849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256688 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7h4l\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f07ac95b-22f0-41cc-8d29-c68702c78185-cert\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxpj\" (UniqueName: \"kubernetes.io/projected/e043eb7e-fee6-4196-b1e7-73c040138f3a-kube-api-access-kjxpj\") pod \"network-check-source-989f8b8dc-6znrp\" (UID: \"e043eb7e-fee6-4196-b1e7-73c040138f3a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256766 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3af45c32-632e-4dbb-8f65-9a30f443b58f-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.256877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.257049 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.256910 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.257574 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.257405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.261172 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.261248 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261170 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f07ac95b-22f0-41cc-8d29-c68702c78185-cert\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.261248 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.261323 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261292 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:23:51.261479 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261465 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:23:51.261690 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.261672 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:23:51.264097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.264081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 16:23:51.266468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.266277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 16:23:51.266468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.266302 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkpxx\"" Jul 07 16:23:51.266468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.266360 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sncwv\"" Jul 07 16:23:51.268542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.268522 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxpj\" (UniqueName: \"kubernetes.io/projected/e043eb7e-fee6-4196-b1e7-73c040138f3a-kube-api-access-kjxpj\") pod \"network-check-source-989f8b8dc-6znrp\" (UID: \"e043eb7e-fee6-4196-b1e7-73c040138f3a\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" Jul 07 16:23:51.268897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.268880 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7h4l\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.269860 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.269824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.270347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.270331 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrxh\" (UniqueName: \"kubernetes.io/projected/f07ac95b-22f0-41cc-8d29-c68702c78185-kube-api-access-xhrxh\") pod \"ingress-canary-vljgp\" (UID: \"f07ac95b-22f0-41cc-8d29-c68702c78185\") " pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.275155 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.275139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwtz\" (UniqueName: \"kubernetes.io/projected/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-kube-api-access-jgwtz\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.322827 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.322806 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" Jul 07 16:23:51.344635 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.344615 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vljgp" Jul 07 16:23:51.484714 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.484678 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp"] Jul 07 16:23:51.488874 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:51.488821 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode043eb7e_fee6_4196_b1e7_73c040138f3a.slice/crio-bc8b895ec4ff39f413ee849bdb401ff5c771e8655feb11e8f0c59f6f59a3cca6 WatchSource:0}: Error finding container bc8b895ec4ff39f413ee849bdb401ff5c771e8655feb11e8f0c59f6f59a3cca6: Status 404 returned error can't find the container with id bc8b895ec4ff39f413ee849bdb401ff5c771e8655feb11e8f0c59f6f59a3cca6 Jul 07 16:23:51.504976 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.504956 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vljgp"] Jul 07 16:23:51.508068 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:23:51.508044 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07ac95b_22f0_41cc_8d29_c68702c78185.slice/crio-65cfd86b8231ff70b557c3680cf9e2cc7ac9b66b0d14b9ac3f07d21b76cf679c WatchSource:0}: Error finding container 65cfd86b8231ff70b557c3680cf9e2cc7ac9b66b0d14b9ac3f07d21b76cf679c: Status 404 returned error can't find the container with id 65cfd86b8231ff70b557c3680cf9e2cc7ac9b66b0d14b9ac3f07d21b76cf679c Jul 07 16:23:51.760177 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.760113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:51.760177 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.760165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:51.760196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760275 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760289 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760293 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760330 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760342 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:52.760329678 +0000 UTC m=+37.069168934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:23:51.760354 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760356 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:52.760350319 +0000 UTC m=+37.069189576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:23:51.760680 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:51.760377 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:52.760363976 +0000 UTC m=+37.069203237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:23:52.436875 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:52.436821 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vljgp" event={"ID":"f07ac95b-22f0-41cc-8d29-c68702c78185","Type":"ContainerStarted","Data":"65cfd86b8231ff70b557c3680cf9e2cc7ac9b66b0d14b9ac3f07d21b76cf679c"} Jul 07 16:23:52.438228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:52.438201 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" event={"ID":"e043eb7e-fee6-4196-b1e7-73c040138f3a","Type":"ContainerStarted","Data":"bc8b895ec4ff39f413ee849bdb401ff5c771e8655feb11e8f0c59f6f59a3cca6"} Jul 07 16:23:52.768890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:52.768798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:52.769032 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:52.768888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:52.769032 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.768979 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:23:52.769032 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:52.769020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.768979 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.769048 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:54.769028401 +0000 UTC m=+39.077867669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.769088 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.769097 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:54.769087078 +0000 UTC m=+39.077926340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.769101 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:23:52.769186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:52.769136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:54.769125373 +0000 UTC m=+39.077964632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:23:53.391202 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:53.391115 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:53.502496 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:53.502461 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:54.787318 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:54.787290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:54.787342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787436 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787456 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787483 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787493 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:58.787475957 +0000 UTC m=+43.096315214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:54.787514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787538 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:23:58.787517975 +0000 UTC m=+43.096357243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787564 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:23:54.787746 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:54.787599 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:23:58.787586774 +0000 UTC m=+43.096426032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:23:55.453348 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:55.453297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:57.383179 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.383148 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:57.452956 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.452927 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="2d2d4c5fecaccdae88aa3c436e75383a5f470ea3175362f7708179ff0896cdbb" exitCode=0 Jul 07 16:23:57.453048 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.453014 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"2d2d4c5fecaccdae88aa3c436e75383a5f470ea3175362f7708179ff0896cdbb"} Jul 07 16:23:57.457060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.457026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vljgp" event={"ID":"f07ac95b-22f0-41cc-8d29-c68702c78185","Type":"ContainerStarted","Data":"7a9c5f72c46ac5a8ce4eab2629f1e7f6f2e315a4ec03d2e5535f2a2f39086b7c"} Jul 07 16:23:57.458874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.458851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" event={"ID":"e043eb7e-fee6-4196-b1e7-73c040138f3a","Type":"ContainerStarted","Data":"f03f61d4a4d1ec21474bffe3f1d53fe1d7ee4de2dfa3fb3deceb455c2b30b0ca"} Jul 07 16:23:57.498899 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.498871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:57.512107 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.512062 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vljgp" podStartSLOduration=1.046848506 podStartE2EDuration="6.512049806s" totalImagesPullingTime="5.4652013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:57.511349288 +0000 UTC m=+41.820188567" watchObservedRunningTime="2026-07-07 16:23:57.512049806 +0000 UTC m=+41.820889066" Jul 07 16:23:57.533064 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.533029 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-6znrp" podStartSLOduration=36.048665694 podStartE2EDuration="41.533015455s" totalImagesPullingTime="5.484349761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:57.532361278 +0000 UTC m=+41.841200558" watchObservedRunningTime="2026-07-07 16:23:57.533015455 +0000 UTC m=+41.841854733" Jul 07 16:23:57.883237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.883206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:57.978720 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:57.978666 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:58.464488 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:58.464453 2569 generic.go:356] "Generic (PLEG): container finished" podID="cfb4efee-7e9b-42dd-b216-07f03d7365e9" containerID="3debcb6e31f53089e66bb3c38c61820a7f50532fd863a27b843020b0263beebf" exitCode=0 Jul 07 16:23:58.464821 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:58.464494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerDied","Data":"3debcb6e31f53089e66bb3c38c61820a7f50532fd863a27b843020b0263beebf"} Jul 07 16:23:58.814498 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:58.814435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:23:58.814498 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:58.814479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:58.814517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814592 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814612 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814623 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814635 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:23:58.814670 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814661 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:06.814639394 +0000 UTC m=+51.123478656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:23:58.814903 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814681 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:06.81467185 +0000 UTC m=+51.123511106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:23:58.814903 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:23:58.814696 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:24:06.814687855 +0000 UTC m=+51.123527130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:23:59.471034 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:59.470998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" event={"ID":"cfb4efee-7e9b-42dd-b216-07f03d7365e9","Type":"ContainerStarted","Data":"d430c9603935082e232fd350ddb3b459ac0aa32fd8d0b67051c2aa15409698d8"} Jul 07 16:23:59.496828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:59.496780 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kwdg7" podStartSLOduration=7.741630739 podStartE2EDuration="42.496764874s" totalImagesPullingTime="34.755134135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:23:59.495002736 +0000 UTC m=+43.803842013" watchObservedRunningTime="2026-07-07 16:23:59.496764874 +0000 UTC m=+43.805604152" Jul 07 16:23:59.618006 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:59.617978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:59.717778 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:23:59.717744 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:00.414585 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:00.414551 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:00.474636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:00.474611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:00.542298 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:00.542263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:02.091176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:02.091118 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:02.167391 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:02.167353 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:04.626278 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:04.626235 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:04.684106 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:04.684071 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:06.867581 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:06.867539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:06.867594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:06.867639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867700 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867744 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867749 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867767 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867778 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:24:22.867755166 +0000 UTC m=+67.176594423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867806 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:22.867790982 +0000 UTC m=+67.176630239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:24:06.868011 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:06.867823 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:22.867815209 +0000 UTC m=+67.176654465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:24:17.448782 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:17.448739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wm9gg" Jul 07 16:24:17.706218 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:17.706106 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:17.778066 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:17.778033 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:19.881894 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:19.881848 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:19.983068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:19.983036 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:21.968281 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.968252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:24:21.968744 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.968298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:24:21.968744 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.968446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:24:21.971654 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.971630 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5z9g\" (UniqueName: \"kubernetes.io/projected/d20d6aed-2688-4b41-bc07-80774e325072-kube-api-access-j5z9g\") pod \"network-check-target-7gkjz\" (UID: \"d20d6aed-2688-4b41-bc07-80774e325072\") " pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:24:21.973586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.973568 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 16:24:21.973799 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.973782 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 16:24:21.979306 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:21.979284 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 16:24:21.979432 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:21.979341 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:25:25.979324285 +0000 UTC m=+130.288163541 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : secret "metrics-daemon-secret" not found Jul 07 16:24:21.981428 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:21.981405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f6620e90-14c4-4008-a09e-d56578336e2a-original-pull-secret\") pod \"global-pull-secret-syncer-nk2sf\" (UID: \"f6620e90-14c4-4008-a09e-d56578336e2a\") " pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:24:22.189909 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.189883 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk2sf" Jul 07 16:24:22.205303 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.205276 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkpxx\"" Jul 07 16:24:22.212431 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.212412 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:24:22.324606 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.324582 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk2sf"] Jul 07 16:24:22.327049 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:24:22.327020 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6620e90_14c4_4008_a09e_d56578336e2a.slice/crio-efa106d6f1302ffafc6ee317a656411ec3828b0729b97b0af5dce27ebcc4438a WatchSource:0}: Error finding container efa106d6f1302ffafc6ee317a656411ec3828b0729b97b0af5dce27ebcc4438a: Status 404 returned error can't find the container with id efa106d6f1302ffafc6ee317a656411ec3828b0729b97b0af5dce27ebcc4438a Jul 07 16:24:22.341814 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.341793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gkjz"] Jul 07 16:24:22.344429 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:24:22.344407 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20d6aed_2688_4b41_bc07_80774e325072.slice/crio-3ef148356aa0870e74c44adb678a67abb2a7a1778924b4b131ff1de6ac4c04f9 WatchSource:0}: Error finding container 3ef148356aa0870e74c44adb678a67abb2a7a1778924b4b131ff1de6ac4c04f9: Status 404 returned error can't find the container with id 3ef148356aa0870e74c44adb678a67abb2a7a1778924b4b131ff1de6ac4c04f9 Jul 07 16:24:22.400154 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.400132 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:22.500989 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.500934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:22.526972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.526943 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gkjz" event={"ID":"d20d6aed-2688-4b41-bc07-80774e325072","Type":"ContainerStarted","Data":"be327d73fda045ea62f54066ac8d595e2e241a54605fd1b787c8e0e32d40cf91"} Jul 07 16:24:22.527082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.527002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:24:22.527082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.527027 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gkjz" event={"ID":"d20d6aed-2688-4b41-bc07-80774e325072","Type":"ContainerStarted","Data":"3ef148356aa0870e74c44adb678a67abb2a7a1778924b4b131ff1de6ac4c04f9"} Jul 07 16:24:22.528141 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.528121 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk2sf" event={"ID":"f6620e90-14c4-4008-a09e-d56578336e2a","Type":"ContainerStarted","Data":"efa106d6f1302ffafc6ee317a656411ec3828b0729b97b0af5dce27ebcc4438a"} Jul 07 16:24:22.544081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.544037 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7gkjz" podStartSLOduration=66.544021702 podStartE2EDuration="1m6.544021702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:22.543574385 +0000 UTC m=+66.852413664" watchObservedRunningTime="2026-07-07 16:24:22.544021702 +0000 UTC m=+66.852860981" Jul 07 16:24:22.601730 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.601696 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:22.873624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.873602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:24:22.873758 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.873641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:24:22.873758 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:22.873675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873765 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873772 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873769 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873821 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873813 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:24:54.873801468 +0000 UTC m=+99.182640727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873878 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:54.873864816 +0000 UTC m=+99.182704077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:24:22.873900 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:22.873893 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:24:54.873884637 +0000 UTC m=+99.182723905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:24:27.542046 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:27.541959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk2sf" event={"ID":"f6620e90-14c4-4008-a09e-d56578336e2a","Type":"ContainerStarted","Data":"61b999565f373a677057aca354188fb8656882f4323de956705e1c577bba97c5"} Jul 07 16:24:27.564128 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:27.564087 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nk2sf" podStartSLOduration=65.724832206 podStartE2EDuration="1m10.564074425s" totalImagesPullingTime="4.839242219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:27.563777282 +0000 UTC m=+71.872616560" watchObservedRunningTime="2026-07-07 16:24:27.564074425 +0000 UTC m=+71.872913700" Jul 07 16:24:31.290698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:31.290641 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:31.361036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:31.360993 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:45.565751 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:45.565586 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:53.533721 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:53.533681 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7gkjz" Jul 07 16:24:54.886440 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:54.886406 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:24:54.886440 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:54.886455 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886550 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886566 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:24:54.886591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886623 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:25:58.886606774 +0000 UTC m=+163.195446032 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886669 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886680 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886683 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:25:58.88665799 +0000 UTC m=+163.195497260 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:24:54.886931 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:24:54.886735 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:25:58.886719935 +0000 UTC m=+163.195559191 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:25:06.466910 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:06.466866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:25:25.998893 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:25.998826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:25:25.999402 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:25.998973 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 16:25:25.999402 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:25.999052 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:27.999034913 +0000 UTC m=+252.307874169 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : secret "metrics-daemon-secret" not found Jul 07 16:25:54.013712 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:54.013664 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" podUID="2f155471-7d49-4047-bcba-9647f0016638" Jul 07 16:25:54.029828 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:54.029792 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" podUID="3af45c32-632e-4dbb-8f65-9a30f443b58f" Jul 07 16:25:54.039996 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:54.039963 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7htvg" podUID="7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4" Jul 07 16:25:54.298454 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:54.298378 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:25:54.757409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:54.757385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:25:54.757547 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:54.757387 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:25:54.757624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:54.757392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7htvg" Jul 07 16:25:58.909384 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:58.909344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:58.909397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:25:58.909434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909495 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909545 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909581 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls podName:7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:00.909564248 +0000 UTC m=+285.218403508 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls") pod "dns-default-7htvg" (UID: "7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4") : secret "dns-default-metrics-tls" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909621 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:28:00.909602219 +0000 UTC m=+285.218441480 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909548 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909639 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85dfbb85f8-tcpsz: secret "image-registry-tls" not found Jul 07 16:25:58.909789 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:25:58.909687 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls podName:2f155471-7d49-4047-bcba-9647f0016638 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:00.909679112 +0000 UTC m=+285.218518368 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls") pod "image-registry-85dfbb85f8-tcpsz" (UID: "2f155471-7d49-4047-bcba-9647f0016638") : secret "image-registry-tls" not found Jul 07 16:26:09.261399 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:26:09.261299 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:27:13.132450 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:13.132414 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:28.081552 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:28.081519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:27:28.081986 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:28.081669 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 16:27:28.081986 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:28.081733 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs podName:0d189f61-09ee-44d8-a2e0-38b22a416ad4 nodeName:}" failed. No retries permitted until 2026-07-07 16:29:30.081716336 +0000 UTC m=+374.390555591 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs") pod "network-metrics-daemon-cvcbn" (UID: "0d189f61-09ee-44d8-a2e0-38b22a416ad4") : secret "metrics-daemon-secret" not found Jul 07 16:27:42.566212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.566118 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v"] Jul 07 16:27:42.568164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.568139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.573495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.573465 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-prqd8"] Jul 07 16:27:42.573856 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.573812 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 16:27:42.574048 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.574025 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 16:27:42.574516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.574492 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 16:27:42.574968 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.574943 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9p2gt\"" Jul 07 16:27:42.576015 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.575811 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7"] Jul 07 16:27:42.576107 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.576059 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.577962 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.577935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.583208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583182 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:27:42.583328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583183 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 16:27:42.583500 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583467 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:27:42.583591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583476 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 16:27:42.583591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583569 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 16:27:42.583798 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583769 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zvwck\"" Jul 07 16:27:42.583993 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.583968 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5v7hw\"" Jul 07 16:27:42.584257 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.584228 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 16:27:42.584385 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.584322 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v"] Jul 07 16:27:42.584807 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.584780 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 16:27:42.587944 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.587923 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:27:42.595151 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.595126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 16:27:42.600056 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.599972 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7"] Jul 07 16:27:42.605462 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.605436 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-prqd8"] Jul 07 16:27:42.669799 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.669773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-serving-cert\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.669915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.669808 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8qj\" (UniqueName: \"kubernetes.io/projected/6af56460-28e6-42ac-82d3-b3240f3eaa53-kube-api-access-dw8qj\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.669987 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.669923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.669987 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.669957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-config\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.669987 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.669977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-trusted-ca\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.670145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.670046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.670145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.670109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.670394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.670374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54nw\" (UniqueName: \"kubernetes.io/projected/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-kube-api-access-r54nw\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.670477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.670407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfvn\" (UniqueName: \"kubernetes.io/projected/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-kube-api-access-flfvn\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.670732 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.670714 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg"] Jul 07 16:27:42.672490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.672472 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-zmhwf"] Jul 07 16:27:42.672598 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.672583 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" Jul 07 16:27:42.674134 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.674116 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.675081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.675063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:27:42.675081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.675074 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 16:27:42.675611 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.675594 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5c85f\"" Jul 07 16:27:42.677735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.677715 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 16:27:42.682746 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.682718 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 16:27:42.682848 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.682725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 16:27:42.683522 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.683499 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt6bc\"" Jul 07 16:27:42.684297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.684009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 16:27:42.688707 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.688690 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 16:27:42.695477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.695456 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-zmhwf"] Jul 07 16:27:42.696480 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.696460 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg"] Jul 07 16:27:42.771307 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771404 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771450 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9be938-cda2-46fb-bab1-87b3b8584302-serving-cert\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771508 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771484 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8qj\" (UniqueName: \"kubernetes.io/projected/6af56460-28e6-42ac-82d3-b3240f3eaa53-kube-api-access-dw8qj\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.771620 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-serving-cert\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.771620 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.771734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-trusted-ca\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.771734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.771734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.771734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-tmp\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lch96\" (UniqueName: \"kubernetes.io/projected/9b9be938-cda2-46fb-bab1-87b3b8584302-kube-api-access-lch96\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:42.771767 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-snapshots\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:42.771828 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls podName:6af56460-28e6-42ac-82d3-b3240f3eaa53 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:43.271809927 +0000 UTC m=+267.580649184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2lkg7" (UID: "6af56460-28e6-42ac-82d3-b3240f3eaa53") : secret "samples-operator-tls" not found Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfvn\" (UniqueName: \"kubernetes.io/projected/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-kube-api-access-flfvn\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54nw\" (UniqueName: \"kubernetes.io/projected/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-kube-api-access-r54nw\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.771966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl69t\" (UniqueName: \"kubernetes.io/projected/f01eb24c-5045-42d7-9663-2e4724d5b361-kube-api-access-xl69t\") pod \"volume-data-source-validator-5d456fdbcd-rwpxg\" (UID: \"f01eb24c-5045-42d7-9663-2e4724d5b361\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" Jul 07 16:27:42.772310 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.771979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-config\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.772371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.772312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.772587 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.772567 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-config\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.772647 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.772572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-trusted-ca\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.773819 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.773793 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-serving-cert\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.774022 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.774001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.781450 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.781426 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8qj\" (UniqueName: \"kubernetes.io/projected/6af56460-28e6-42ac-82d3-b3240f3eaa53-kube-api-access-dw8qj\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:42.781543 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.781464 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfvn\" (UniqueName: \"kubernetes.io/projected/a1983aed-25a3-4fb3-b9cc-08ebdbbd7691-kube-api-access-flfvn\") pod \"kube-storage-version-migrator-operator-846bf668f7-v7n9v\" (UID: \"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.781946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.781929 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54nw\" (UniqueName: \"kubernetes.io/projected/3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f-kube-api-access-r54nw\") pod \"console-operator-68b757fb67-prqd8\" (UID: \"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f\") " pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.873228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-tmp\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lch96\" (UniqueName: \"kubernetes.io/projected/9b9be938-cda2-46fb-bab1-87b3b8584302-kube-api-access-lch96\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-snapshots\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl69t\" (UniqueName: \"kubernetes.io/projected/f01eb24c-5045-42d7-9663-2e4724d5b361-kube-api-access-xl69t\") pod \"volume-data-source-validator-5d456fdbcd-rwpxg\" (UID: \"f01eb24c-5045-42d7-9663-2e4724d5b361\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" Jul 07 16:27:42.873389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9be938-cda2-46fb-bab1-87b3b8584302-serving-cert\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873616 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873596 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-tmp\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873854 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873816 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9b9be938-cda2-46fb-bab1-87b3b8584302-snapshots\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.873915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.873861 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.874136 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.874119 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b9be938-cda2-46fb-bab1-87b3b8584302-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.875211 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.875194 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9be938-cda2-46fb-bab1-87b3b8584302-serving-cert\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.879174 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.879158 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" Jul 07 16:27:42.886064 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.886043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl69t\" (UniqueName: \"kubernetes.io/projected/f01eb24c-5045-42d7-9663-2e4724d5b361-kube-api-access-xl69t\") pod \"volume-data-source-validator-5d456fdbcd-rwpxg\" (UID: \"f01eb24c-5045-42d7-9663-2e4724d5b361\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" Jul 07 16:27:42.886202 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.886185 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lch96\" (UniqueName: \"kubernetes.io/projected/9b9be938-cda2-46fb-bab1-87b3b8584302-kube-api-access-lch96\") pod \"insights-operator-5c847dcfc4-zmhwf\" (UID: \"9b9be938-cda2-46fb-bab1-87b3b8584302\") " pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:42.889993 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.889973 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:42.946131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.946078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:42.982966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.982939 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" Jul 07 16:27:42.989076 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:42.989044 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" Jul 07 16:27:43.011077 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.011049 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v"] Jul 07 16:27:43.014868 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:43.014820 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1983aed_25a3_4fb3_b9cc_08ebdbbd7691.slice/crio-4f677855217964880823719af17d45f4fa2cffd0825939a77e8ec0b6cd64d17a WatchSource:0}: Error finding container 4f677855217964880823719af17d45f4fa2cffd0825939a77e8ec0b6cd64d17a: Status 404 returned error can't find the container with id 4f677855217964880823719af17d45f4fa2cffd0825939a77e8ec0b6cd64d17a Jul 07 16:27:43.030549 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.030528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-prqd8"] Jul 07 16:27:43.033703 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:43.033653 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb82cc1_986b_4f4a_bfab_ff9c2dcd836f.slice/crio-7754096b06e135b16b55fcdc7b34931908a8e0cd357da8efa663f77bfb830daa WatchSource:0}: Error finding container 7754096b06e135b16b55fcdc7b34931908a8e0cd357da8efa663f77bfb830daa: Status 404 returned error can't find the container with id 7754096b06e135b16b55fcdc7b34931908a8e0cd357da8efa663f77bfb830daa Jul 07 16:27:43.037462 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.037259 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:43.115221 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.115184 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg"] Jul 07 16:27:43.118221 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:43.118195 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01eb24c_5045_42d7_9663_2e4724d5b361.slice/crio-eaa8622fb3914a7dbcf21a8778ad3d44ff1683aa7923eccac670005d0162675a WatchSource:0}: Error finding container eaa8622fb3914a7dbcf21a8778ad3d44ff1683aa7923eccac670005d0162675a: Status 404 returned error can't find the container with id eaa8622fb3914a7dbcf21a8778ad3d44ff1683aa7923eccac670005d0162675a Jul 07 16:27:43.134146 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.134096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-zmhwf"] Jul 07 16:27:43.136775 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:43.136756 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9be938_cda2_46fb_bab1_87b3b8584302.slice/crio-d5c950faf82d7527c3cf66e2851f0a3ce51e45079a58706b33aa4016a7aaa5bc WatchSource:0}: Error finding container d5c950faf82d7527c3cf66e2851f0a3ce51e45079a58706b33aa4016a7aaa5bc: Status 404 returned error can't find the container with id d5c950faf82d7527c3cf66e2851f0a3ce51e45079a58706b33aa4016a7aaa5bc Jul 07 16:27:43.275943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:43.275910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:43.276097 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:43.276077 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 16:27:43.276164 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:43.276148 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls podName:6af56460-28e6-42ac-82d3-b3240f3eaa53 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:44.276127906 +0000 UTC m=+268.584967180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2lkg7" (UID: "6af56460-28e6-42ac-82d3-b3240f3eaa53") : secret "samples-operator-tls" not found Jul 07 16:27:44.020974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:44.020934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" event={"ID":"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f","Type":"ContainerStarted","Data":"7754096b06e135b16b55fcdc7b34931908a8e0cd357da8efa663f77bfb830daa"} Jul 07 16:27:44.022317 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:44.022256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" event={"ID":"9b9be938-cda2-46fb-bab1-87b3b8584302","Type":"ContainerStarted","Data":"d5c950faf82d7527c3cf66e2851f0a3ce51e45079a58706b33aa4016a7aaa5bc"} Jul 07 16:27:44.023532 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:44.023466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" event={"ID":"f01eb24c-5045-42d7-9663-2e4724d5b361","Type":"ContainerStarted","Data":"eaa8622fb3914a7dbcf21a8778ad3d44ff1683aa7923eccac670005d0162675a"} Jul 07 16:27:44.024615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:44.024591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" event={"ID":"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691","Type":"ContainerStarted","Data":"4f677855217964880823719af17d45f4fa2cffd0825939a77e8ec0b6cd64d17a"} Jul 07 16:27:44.286584 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:44.286192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:44.286584 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:44.286367 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 16:27:44.286584 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:44.286430 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls podName:6af56460-28e6-42ac-82d3-b3240f3eaa53 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:46.286411233 +0000 UTC m=+270.595250490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2lkg7" (UID: "6af56460-28e6-42ac-82d3-b3240f3eaa53") : secret "samples-operator-tls" not found Jul 07 16:27:45.029158 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:45.029117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" event={"ID":"f01eb24c-5045-42d7-9663-2e4724d5b361","Type":"ContainerStarted","Data":"198453f8247dd8c4092baf7c2d24effe010d67ae9c0134bed655a602386fbd44"} Jul 07 16:27:45.052562 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:45.052511 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rwpxg" podStartSLOduration=1.808671868 podStartE2EDuration="3.052491791s" totalImagesPullingTime="1.243819923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:27:45.051904784 +0000 UTC m=+269.360744064" watchObservedRunningTime="2026-07-07 16:27:45.052491791 +0000 UTC m=+269.361331070" Jul 07 16:27:45.166139 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:45.166106 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:45.309646 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:45.309576 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:46.302195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:46.302107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:46.302642 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:46.302298 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 16:27:46.302642 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:46.302376 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls podName:6af56460-28e6-42ac-82d3-b3240f3eaa53 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:50.302354417 +0000 UTC m=+274.611193690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2lkg7" (UID: "6af56460-28e6-42ac-82d3-b3240f3eaa53") : secret "samples-operator-tls" not found Jul 07 16:27:47.036575 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.036545 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/0.log" Jul 07 16:27:47.036735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.036593 2569 generic.go:356] "Generic (PLEG): container finished" podID="3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f" containerID="2ca61c65320f5ebeb52796c79cebc0783d84afc543e2b4f6b913d1b20aca3595" exitCode=255 Jul 07 16:27:47.036735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.036674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" event={"ID":"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f","Type":"ContainerDied","Data":"2ca61c65320f5ebeb52796c79cebc0783d84afc543e2b4f6b913d1b20aca3595"} Jul 07 16:27:47.036881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.036863 2569 scope.go:117] "RemoveContainer" containerID="2ca61c65320f5ebeb52796c79cebc0783d84afc543e2b4f6b913d1b20aca3595" Jul 07 16:27:47.038266 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.038242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" event={"ID":"9b9be938-cda2-46fb-bab1-87b3b8584302","Type":"ContainerStarted","Data":"b2dedacb8e64ca5d6c3b9e31c19bc7aeb1f905796370ee8e9c5592257201bf72"} Jul 07 16:27:47.039719 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.039699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" event={"ID":"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691","Type":"ContainerStarted","Data":"ce0c35356ab1a39e27c260ee05041f6443a01a5df58165adbefa9ff4699f1250"} Jul 07 16:27:47.088947 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.088902 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" podStartSLOduration=1.91867593 podStartE2EDuration="5.088884273s" totalImagesPullingTime="3.170208343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:27:47.088131294 +0000 UTC m=+271.396970573" watchObservedRunningTime="2026-07-07 16:27:47.088884273 +0000 UTC m=+271.397723551" Jul 07 16:27:47.122406 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.122354 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" podStartSLOduration=2.070184165 podStartE2EDuration="5.122332683s" totalImagesPullingTime="3.052148518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:27:47.120575818 +0000 UTC m=+271.429415098" watchObservedRunningTime="2026-07-07 16:27:47.122332683 +0000 UTC m=+271.431171963" Jul 07 16:27:47.655449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.655421 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:47.809637 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.809611 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:47.859166 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.859142 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96"] Jul 07 16:27:47.861066 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.861052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" Jul 07 16:27:47.864106 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.864086 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 16:27:47.864217 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.864109 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 16:27:47.865081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.865066 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dmfgg\"" Jul 07 16:27:47.884445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.884419 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96"] Jul 07 16:27:47.998415 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:47.998358 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:48.016932 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.016910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qt4s\" (UniqueName: \"kubernetes.io/projected/b705125e-6205-4969-94fa-92aa8e6dafd3-kube-api-access-5qt4s\") pod \"migrator-d57d58cc9-4gs96\" (UID: \"b705125e-6205-4969-94fa-92aa8e6dafd3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" Jul 07 16:27:48.044205 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044186 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:27:48.044586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044568 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/0.log" Jul 07 16:27:48.044674 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044600 2569 generic.go:356] "Generic (PLEG): container finished" podID="3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f" containerID="305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8" exitCode=255 Jul 07 16:27:48.044674 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044667 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" event={"ID":"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f","Type":"ContainerDied","Data":"305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8"} Jul 07 16:27:48.044751 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044704 2569 scope.go:117] "RemoveContainer" containerID="2ca61c65320f5ebeb52796c79cebc0783d84afc543e2b4f6b913d1b20aca3595" Jul 07 16:27:48.044938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.044919 2569 scope.go:117] "RemoveContainer" containerID="305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8" Jul 07 16:27:48.045132 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.045109 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-prqd8_openshift-console-operator(3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f)\"" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" podUID="3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f" Jul 07 16:27:48.117358 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.117338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qt4s\" (UniqueName: \"kubernetes.io/projected/b705125e-6205-4969-94fa-92aa8e6dafd3-kube-api-access-5qt4s\") pod \"migrator-d57d58cc9-4gs96\" (UID: \"b705125e-6205-4969-94fa-92aa8e6dafd3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" Jul 07 16:27:48.133434 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.133418 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qt4s\" (UniqueName: \"kubernetes.io/projected/b705125e-6205-4969-94fa-92aa8e6dafd3-kube-api-access-5qt4s\") pod \"migrator-d57d58cc9-4gs96\" (UID: \"b705125e-6205-4969-94fa-92aa8e6dafd3\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" Jul 07 16:27:48.169785 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.169764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" Jul 07 16:27:48.194090 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.192777 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mk524"] Jul 07 16:27:48.196057 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.196036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.199915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.199892 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:48.204282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.204259 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2w8q7\"" Jul 07 16:27:48.206337 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.206314 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 16:27:48.216093 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.215874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 16:27:48.217186 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.217166 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mk524"] Jul 07 16:27:48.319105 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.319072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqzs\" (UniqueName: \"kubernetes.io/projected/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-api-access-nkqzs\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.319232 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.319137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.319232 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.319202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-crio-socket\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.319302 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.319248 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.319302 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.319276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-data-volume\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.344083 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:48.344054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb705125e_6205_4969_94fa_92aa8e6dafd3.slice/crio-c5498b5e39dbe61344209e8668698a60108d9b1071437bd1b460ff45ebf11818 WatchSource:0}: Error finding container c5498b5e39dbe61344209e8668698a60108d9b1071437bd1b460ff45ebf11818: Status 404 returned error can't find the container with id c5498b5e39dbe61344209e8668698a60108d9b1071437bd1b460ff45ebf11818 Jul 07 16:27:48.345141 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.345121 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96"] Jul 07 16:27:48.420222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqzs\" (UniqueName: \"kubernetes.io/projected/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-api-access-nkqzs\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420314 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420314 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-crio-socket\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420314 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-data-volume\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420392 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-crio-socket\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.420458 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.420417 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 16:27:48.420458 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.420439 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:27:48.420596 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.420480 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:48.920461438 +0000 UTC m=+273.229300708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : configmap "kube-rbac-proxy" not found Jul 07 16:27:48.420596 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.420499 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:48.920489261 +0000 UTC m=+273.229328520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:27:48.420668 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.420599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-data-volume\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.432464 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.432440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqzs\" (UniqueName: \"kubernetes.io/projected/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-api-access-nkqzs\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.485595 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.485567 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:48.585957 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.585923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:48.924252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.924186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.924252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:48.924227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:48.924631 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.924324 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 16:27:48.924631 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.924357 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:27:48.924631 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.924383 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:49.92437066 +0000 UTC m=+274.233209917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : configmap "kube-rbac-proxy" not found Jul 07 16:27:48.924631 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:48.924422 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:49.924403158 +0000 UTC m=+274.233242414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:27:49.048929 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.048899 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" event={"ID":"b705125e-6205-4969-94fa-92aa8e6dafd3","Type":"ContainerStarted","Data":"c5498b5e39dbe61344209e8668698a60108d9b1071437bd1b460ff45ebf11818"} Jul 07 16:27:49.050650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.050630 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:27:49.051034 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.051015 2569 scope.go:117] "RemoveContainer" containerID="305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8" Jul 07 16:27:49.051224 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:49.051204 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-prqd8_openshift-console-operator(3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f)\"" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" podUID="3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f" Jul 07 16:27:49.931778 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.931756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:49.932145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.931798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:49.932145 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:49.931912 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 16:27:49.932145 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:49.931944 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:27:49.932145 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:49.931975 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:51.931954339 +0000 UTC m=+276.240793603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : configmap "kube-rbac-proxy" not found Jul 07 16:27:49.932145 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:49.932000 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:51.931982925 +0000 UTC m=+276.240822198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:27:49.981260 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.980560 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tnkj7"] Jul 07 16:27:49.982734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.982715 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:49.987568 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.987548 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 16:27:49.987795 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.987777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 16:27:49.987874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.987827 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 16:27:49.988006 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.987979 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 16:27:49.988601 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.988583 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-s25rt\"" Jul 07 16:27:49.997817 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:49.997796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tnkj7"] Jul 07 16:27:50.035166 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.035125 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:50.056048 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.056025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" event={"ID":"b705125e-6205-4969-94fa-92aa8e6dafd3","Type":"ContainerStarted","Data":"0804d711230bcb382b4d6db033d8d29aa87796efbccd1e187444d6bd41a5a73d"} Jul 07 16:27:50.056154 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.056068 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" event={"ID":"b705125e-6205-4969-94fa-92aa8e6dafd3","Type":"ContainerStarted","Data":"482a061b0662b8798a1c9a94d3ef0952235c078a1fe9dc397e04e00b67f994ff"} Jul 07 16:27:50.076169 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.076126 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4gs96" podStartSLOduration=1.549747629 podStartE2EDuration="3.07611319s" totalImagesPullingTime="1.526365561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:27:50.075039323 +0000 UTC m=+274.383878603" watchObservedRunningTime="2026-07-07 16:27:50.07611319 +0000 UTC m=+274.384952467" Jul 07 16:27:50.133132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.133072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/66114466-1960-4072-afaa-40e02a5c6306-signing-cabundle\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.133132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.133112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/66114466-1960-4072-afaa-40e02a5c6306-kube-api-access-rhpl8\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.133301 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.133183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/66114466-1960-4072-afaa-40e02a5c6306-signing-key\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.233986 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.233960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/66114466-1960-4072-afaa-40e02a5c6306-signing-cabundle\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.234075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.233999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/66114466-1960-4072-afaa-40e02a5c6306-kube-api-access-rhpl8\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.234075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.234032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/66114466-1960-4072-afaa-40e02a5c6306-signing-key\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.234727 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.234710 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/66114466-1960-4072-afaa-40e02a5c6306-signing-cabundle\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.236393 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.236373 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/66114466-1960-4072-afaa-40e02a5c6306-signing-key\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.243290 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.243264 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/66114466-1960-4072-afaa-40e02a5c6306-kube-api-access-rhpl8\") pod \"service-ca-758f599c5-tnkj7\" (UID: \"66114466-1960-4072-afaa-40e02a5c6306\") " pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.291594 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.291572 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-tnkj7" Jul 07 16:27:50.335018 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.334990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:50.335687 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:50.335191 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 16:27:50.335687 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:50.335267 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls podName:6af56460-28e6-42ac-82d3-b3240f3eaa53 nodeName:}" failed. No retries permitted until 2026-07-07 16:27:58.335249325 +0000 UTC m=+282.644088588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-2lkg7" (UID: "6af56460-28e6-42ac-82d3-b3240f3eaa53") : secret "samples-operator-tls" not found Jul 07 16:27:50.414016 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.413954 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-tnkj7"] Jul 07 16:27:50.416697 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:27:50.416676 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66114466_1960_4072_afaa_40e02a5c6306.slice/crio-4976417a8d78427571bcd31617130668ff9b1e22a98663de6e9d94c74ed53f07 WatchSource:0}: Error finding container 4976417a8d78427571bcd31617130668ff9b1e22a98663de6e9d94c74ed53f07: Status 404 returned error can't find the container with id 4976417a8d78427571bcd31617130668ff9b1e22a98663de6e9d94c74ed53f07 Jul 07 16:27:50.987496 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:50.987468 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5fq8_c9a61f94-1bf2-4834-b9d3-e90c846494c1/dns-node-resolver/0.log" Jul 07 16:27:51.061063 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:51.061032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-tnkj7" event={"ID":"66114466-1960-4072-afaa-40e02a5c6306","Type":"ContainerStarted","Data":"4976417a8d78427571bcd31617130668ff9b1e22a98663de6e9d94c74ed53f07"} Jul 07 16:27:51.846470 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:51.846438 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:51.948055 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:51.948024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:51.948186 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:51.948060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:51.948186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:51.948170 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:27:51.948186 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:51.948179 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 16:27:51.948285 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:51.948229 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:55.948215274 +0000 UTC m=+280.257054533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:27:51.948285 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:51.948244 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:27:55.948237802 +0000 UTC m=+280.257077063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : configmap "kube-rbac-proxy" not found Jul 07 16:27:52.187519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:52.187452 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wnbfm_0e9f2128-48f5-4b83-816b-eba10e7a1bf2/node-ca/0.log" Jul 07 16:27:52.890064 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:52.890030 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:52.890194 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:52.890111 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:27:52.890399 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:52.890387 2569 scope.go:117] "RemoveContainer" containerID="305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8" Jul 07 16:27:52.890535 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:52.890516 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-prqd8_openshift-console-operator(3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f)\"" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" podUID="3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f" Jul 07 16:27:52.986784 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:52.986759 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vljgp_f07ac95b-22f0-41cc-8d29-c68702c78185/serve-healthcheck-canary/0.log" Jul 07 16:27:53.068903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:53.068824 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-tnkj7" event={"ID":"66114466-1960-4072-afaa-40e02a5c6306","Type":"ContainerStarted","Data":"d2bfc329843e43d05d86f9282fc488c87d4015e6089b19329d8c958885dd8710"} Jul 07 16:27:53.098679 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:53.098640 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-tnkj7" podStartSLOduration=1.76243924 podStartE2EDuration="4.098628789s" totalImagesPullingTime="2.336189549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:27:53.097415944 +0000 UTC m=+277.406255222" watchObservedRunningTime="2026-07-07 16:27:53.098628789 +0000 UTC m=+277.407468081" Jul 07 16:27:53.188669 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:53.188625 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4gs96_b705125e-6205-4969-94fa-92aa8e6dafd3/migrator/0.log" Jul 07 16:27:53.386400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:53.386379 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4gs96_b705125e-6205-4969-94fa-92aa8e6dafd3/graceful-termination/0.log" Jul 07 16:27:53.593620 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:53.593598 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-v7n9v_a1983aed-25a3-4fb3-b9cc-08ebdbbd7691/kube-storage-version-migrator-operator/0.log" Jul 07 16:27:54.621414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:54.621381 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:54.825366 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:54.825334 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:55.018491 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:55.018421 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:55.131792 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:55.131759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:55.471656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:55.471630 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:55.977960 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:55.977924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:55.977960 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:55.977963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:27:55.978404 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:55.978068 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 16:27:55.978404 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:55.978131 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:28:03.97811503 +0000 UTC m=+288.286954286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : configmap "kube-rbac-proxy" not found Jul 07 16:27:55.978404 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:55.978069 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:27:55.978404 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:55.978211 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:28:03.978197449 +0000 UTC m=+288.287036705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:27:56.113690 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:56.113660 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:56.701578 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:56.701547 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:56.856785 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:56.856748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:57.284359 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:57.284329 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:57.464916 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:57.464895 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:57.692133 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:57.692112 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:57.757776 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:57.757754 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" podUID="3af45c32-632e-4dbb-8f65-9a30f443b58f" Jul 07 16:27:57.757913 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:57.757898 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7htvg" podUID="7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4" Jul 07 16:27:57.757976 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:27:57.757954 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" podUID="2f155471-7d49-4047-bcba-9647f0016638" Jul 07 16:27:58.084216 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.084193 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:27:58.084555 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.084540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7htvg" Jul 07 16:27:58.084892 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.084877 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:27:58.395109 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.395050 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:58.397442 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.397419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6af56460-28e6-42ac-82d3-b3240f3eaa53-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-2lkg7\" (UID: \"6af56460-28e6-42ac-82d3-b3240f3eaa53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:58.494425 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.494405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" Jul 07 16:27:58.498270 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.498228 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:58.667707 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.667647 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7"] Jul 07 16:27:58.832028 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:58.832003 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:59.014945 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:59.014894 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:59.088759 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:59.088733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" event={"ID":"6af56460-28e6-42ac-82d3-b3240f3eaa53","Type":"ContainerStarted","Data":"f1c58c6c694ff626e049979249bfa7441a6d74968f8fd2209a922cc10e2c5ec7"} Jul 07 16:27:59.184821 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:59.184798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:59.553054 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:59.553019 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:27:59.944869 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:27:59.944821 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:00.514685 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.514648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:00.915451 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.915429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:28:00.915926 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.915467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:28:00.915926 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.915498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:28:00.915926 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:00.915619 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 16:28:00.915926 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:00.915675 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert podName:3af45c32-632e-4dbb-8f65-9a30f443b58f nodeName:}" failed. No retries permitted until 2026-07-07 16:30:02.91565856 +0000 UTC m=+407.224497817 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5qcwh" (UID: "3af45c32-632e-4dbb-8f65-9a30f443b58f") : secret "networking-console-plugin-cert" not found Jul 07 16:28:00.918007 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.917984 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4-metrics-tls\") pod \"dns-default-7htvg\" (UID: \"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4\") " pod="openshift-dns/dns-default-7htvg" Jul 07 16:28:00.918108 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:00.918092 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"image-registry-85dfbb85f8-tcpsz\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:28:01.090174 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.090101 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47hf8\"" Jul 07 16:28:01.091312 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.091297 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7f9fh\"" Jul 07 16:28:01.095166 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.095143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7htvg" Jul 07 16:28:01.095298 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.095221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:28:01.096899 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.096874 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" event={"ID":"6af56460-28e6-42ac-82d3-b3240f3eaa53","Type":"ContainerStarted","Data":"aa7a1c5d9297cdb87741ed7d9b8721a3bc0aa2773e54b4ed8da31b99d5cb6652"} Jul 07 16:28:01.097021 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.096932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" event={"ID":"6af56460-28e6-42ac-82d3-b3240f3eaa53","Type":"ContainerStarted","Data":"0de7308d3b9b85e5b1b92e874838f6f337398750fc29e10eb808655abbab1074"} Jul 07 16:28:01.164122 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.164063 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:01.266302 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.266249 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-2lkg7" podStartSLOduration=17.150288827 podStartE2EDuration="19.266233998s" totalImagesPullingTime="2.115945171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:01.158278485 +0000 UTC m=+285.467117764" watchObservedRunningTime="2026-07-07 16:28:01.266233998 +0000 UTC m=+285.575073296" Jul 07 16:28:01.267178 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.267161 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:28:01.278461 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:01.278433 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f155471_7d49_4047_bcba_9647f0016638.slice/crio-5335ccca0b728301d088d5d072c83830305542fa44ea1d3214d0da1650be44de WatchSource:0}: Error finding container 5335ccca0b728301d088d5d072c83830305542fa44ea1d3214d0da1650be44de: Status 404 returned error can't find the container with id 5335ccca0b728301d088d5d072c83830305542fa44ea1d3214d0da1650be44de Jul 07 16:28:01.279135 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.279113 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7htvg"] Jul 07 16:28:01.282298 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:01.282273 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7310a4c3_ee87_4ac3_ad45_a30fda4fb6f4.slice/crio-25c43962df88b33ffa259bc53dcba8aa79769332a4e17ea5d5b94559d328eb40 WatchSource:0}: Error finding container 25c43962df88b33ffa259bc53dcba8aa79769332a4e17ea5d5b94559d328eb40: Status 404 returned error can't find the container with id 25c43962df88b33ffa259bc53dcba8aa79769332a4e17ea5d5b94559d328eb40 Jul 07 16:28:01.372375 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.372323 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:01.601812 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:01.601778 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:02.100304 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.100270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:02.102109 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.102081 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" event={"ID":"2f155471-7d49-4047-bcba-9647f0016638","Type":"ContainerStarted","Data":"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02"} Jul 07 16:28:02.102235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.102134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" event={"ID":"2f155471-7d49-4047-bcba-9647f0016638","Type":"ContainerStarted","Data":"5335ccca0b728301d088d5d072c83830305542fa44ea1d3214d0da1650be44de"} Jul 07 16:28:02.102235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.102193 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:28:02.103257 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.103233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7htvg" event={"ID":"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4","Type":"ContainerStarted","Data":"25c43962df88b33ffa259bc53dcba8aa79769332a4e17ea5d5b94559d328eb40"} Jul 07 16:28:02.137397 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.137355 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" podStartSLOduration=285.137342501 podStartE2EDuration="4m45.137342501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:02.137112853 +0000 UTC m=+286.445952129" watchObservedRunningTime="2026-07-07 16:28:02.137342501 +0000 UTC m=+286.446181757" Jul 07 16:28:02.744555 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.744521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:02.926057 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:02.926029 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:03.262238 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:03.262209 2569 scope.go:117] "RemoveContainer" containerID="305dca8ad1e97211273025884608d7fdd2477cea826ef5c9609e5ea468160bd8" Jul 07 16:28:03.309878 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:03.309846 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:03.518224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:03.518174 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:03.706107 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:03.706082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:04.040904 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.040877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:04.040904 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.040912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:04.041113 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:04.041021 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 16:28:04.041113 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:04.041080 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls podName:e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df nodeName:}" failed. No retries permitted until 2026-07-07 16:28:20.041065865 +0000 UTC m=+304.349905125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mk524" (UID: "e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df") : secret "insights-runtime-extractor-tls" not found Jul 07 16:28:04.041336 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.041319 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:04.112208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.112190 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:28:04.112324 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.112275 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" event={"ID":"3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f","Type":"ContainerStarted","Data":"f9dc1bbe6344e37f37d3ee61eef4de9e32c8363c22f31ededa6d6f0925a34229"} Jul 07 16:28:04.112611 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.112563 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:28:04.114323 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.114299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7htvg" event={"ID":"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4","Type":"ContainerStarted","Data":"d3fdba6750715a89d16acd8baf2bc36c8b200b4e07fe85d8567aef3414657336"} Jul 07 16:28:04.114416 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.114333 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7htvg" event={"ID":"7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4","Type":"ContainerStarted","Data":"4edaa4ee9dc2e175ae62c7ac91855b85802760bf9ac178ef5fec109c7096e355"} Jul 07 16:28:04.114476 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.114417 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7htvg" Jul 07 16:28:04.139585 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.139541 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" podStartSLOduration=18.981130565 podStartE2EDuration="22.139513082s" totalImagesPullingTime="3.158382517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:04.139230077 +0000 UTC m=+288.448069353" watchObservedRunningTime="2026-07-07 16:28:04.139513082 +0000 UTC m=+288.448352362" Jul 07 16:28:04.168059 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.168019 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7htvg" podStartSLOduration=250.923218674 podStartE2EDuration="4m13.168006352s" totalImagesPullingTime="2.244787678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:04.165013614 +0000 UTC m=+288.473852903" watchObservedRunningTime="2026-07-07 16:28:04.168006352 +0000 UTC m=+288.476845633" Jul 07 16:28:04.236034 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.236001 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-prqd8" Jul 07 16:28:04.567808 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.567777 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:04.854506 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:04.854465 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:05.006272 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:05.006239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:05.167915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:05.167849 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:05.346318 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:05.346290 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.013016 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.012985 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.119700 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.119670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.440165 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.440136 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.581735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.581705 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.782866 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.782787 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:06.954002 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:06.953976 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:07.949208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:07.949170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:12.261926 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:12.261885 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cvcbn" podUID="0d189f61-09ee-44d8-a2e0-38b22a416ad4" Jul 07 16:28:14.120521 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:14.120495 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7htvg" Jul 07 16:28:16.089576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:16.089547 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:28:16.090472 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:16.090446 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:28:16.100585 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:16.100565 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 16:28:17.403694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.403652 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:17.661063 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.660998 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:17.733461 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.733434 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-7h5hw"] Jul 07 16:28:17.746002 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.745972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:17.749935 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.749915 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 16:28:17.750068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.750000 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 16:28:17.750534 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.750515 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9mvwc\"" Jul 07 16:28:17.751770 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.751744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-7h5hw"] Jul 07 16:28:17.788783 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.788749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:17.838107 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.838087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42lv\" (UniqueName: \"kubernetes.io/projected/819f7b33-44b8-4771-8329-e3216cc59113-kube-api-access-s42lv\") pod \"downloads-685bfb5f64-7h5hw\" (UID: \"819f7b33-44b8-4771-8329-e3216cc59113\") " pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:17.938905 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.938829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42lv\" (UniqueName: \"kubernetes.io/projected/819f7b33-44b8-4771-8329-e3216cc59113-kube-api-access-s42lv\") pod \"downloads-685bfb5f64-7h5hw\" (UID: \"819f7b33-44b8-4771-8329-e3216cc59113\") " pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:17.952256 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.952232 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:17.963608 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:17.963584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42lv\" (UniqueName: \"kubernetes.io/projected/819f7b33-44b8-4771-8329-e3216cc59113-kube-api-access-s42lv\") pod \"downloads-685bfb5f64-7h5hw\" (UID: \"819f7b33-44b8-4771-8329-e3216cc59113\") " pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:18.055613 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:18.055595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:18.193634 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:18.193570 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-7h5hw"] Jul 07 16:28:18.197242 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:18.197216 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819f7b33_44b8_4771_8329_e3216cc59113.slice/crio-9d0ab002e5ca66c4595b3ecaf7b2fde05183c6c9d8f08c7d5baca961fccf8700 WatchSource:0}: Error finding container 9d0ab002e5ca66c4595b3ecaf7b2fde05183c6c9d8f08c7d5baca961fccf8700: Status 404 returned error can't find the container with id 9d0ab002e5ca66c4595b3ecaf7b2fde05183c6c9d8f08c7d5baca961fccf8700 Jul 07 16:28:18.199091 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:18.199075 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:28:19.169703 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:19.169664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-7h5hw" event={"ID":"819f7b33-44b8-4771-8329-e3216cc59113","Type":"ContainerStarted","Data":"9d0ab002e5ca66c4595b3ecaf7b2fde05183c6c9d8f08c7d5baca961fccf8700"} Jul 07 16:28:20.013478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.013445 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:20.057212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.057181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:20.059740 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.059717 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mk524\" (UID: \"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df\") " pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:20.118719 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.118694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:20.311624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.311531 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2w8q7\"" Jul 07 16:28:20.317367 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.317339 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mk524" Jul 07 16:28:20.469313 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.469289 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mk524"] Jul 07 16:28:20.471784 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:20.471760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c4b12a_6f4f_4cb6_ba31_5ea4c3a706df.slice/crio-bd71fcd39e62dfff87b0c5161836359445ced33d6cf7acdf912481b44cb20127 WatchSource:0}: Error finding container bd71fcd39e62dfff87b0c5161836359445ced33d6cf7acdf912481b44cb20127: Status 404 returned error can't find the container with id bd71fcd39e62dfff87b0c5161836359445ced33d6cf7acdf912481b44cb20127 Jul 07 16:28:20.645376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.645346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:20.825890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:20.825860 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:21.012896 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:21.012735 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:21.102953 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:21.102913 2569 patch_prober.go:28] interesting pod/image-registry-85dfbb85f8-tcpsz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 16:28:21.103120 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:21.102988 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" podUID="2f155471-7d49-4047-bcba-9647f0016638" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 16:28:21.183041 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:21.182958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk524" event={"ID":"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df","Type":"ContainerStarted","Data":"bd71fcd39e62dfff87b0c5161836359445ced33d6cf7acdf912481b44cb20127"} Jul 07 16:28:23.113930 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:23.113899 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:28:23.585394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:23.585356 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:23.691222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:23.691194 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:24.262122 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:24.262089 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:28:24.373019 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:24.372987 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:24.545829 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:24.545703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:25.200670 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:25.200634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk524" event={"ID":"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df","Type":"ContainerStarted","Data":"0dae2256a48e7bc63e8d7f254cc8ac456e032f4b0e64ca19c38436a8f1673351"} Jul 07 16:28:25.574576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:25.574544 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:26.205607 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:26.205574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk524" event={"ID":"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df","Type":"ContainerStarted","Data":"e162e73ccaaf9ccc6129e9683df5e54e816f697dee871fadab675e244ebe05d7"} Jul 07 16:28:26.280714 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:26.280677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:26.403980 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:26.403947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:26.994436 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:26.994400 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:27.323177 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.323041 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:27.515852 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.515277 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6"] Jul 07 16:28:27.520371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.519028 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc"] Jul 07 16:28:27.520371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.519188 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.522234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.521537 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.533489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.532626 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 16:28:27.533489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.532979 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 16:28:27.533489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.533184 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9s2wp\"" Jul 07 16:28:27.533489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.533348 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 16:28:27.533737 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.533553 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 16:28:27.534517 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.534348 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 16:28:27.537352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.536808 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 16:28:27.537352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.537030 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l8fth\"" Jul 07 16:28:27.537352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.537131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 16:28:27.537352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.537216 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 16:28:27.552440 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.552417 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6"] Jul 07 16:28:27.553692 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.553630 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc"] Jul 07 16:28:27.572431 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.572407 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jtq8q"] Jul 07 16:28:27.575175 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.575127 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.580965 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.580805 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 16:28:27.580965 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.580937 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:27.581178 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.581131 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 16:28:27.581246 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.581190 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jxggr\"" Jul 07 16:28:27.581861 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.581350 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1197d210-3afa-4053-a758-ff0ce4f8e52f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614385 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f92v\" (UniqueName: \"kubernetes.io/projected/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-api-access-8f92v\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hkg\" (UniqueName: \"kubernetes.io/projected/69dc5200-1e97-473d-93a0-a02b6abe5654-kube-api-access-w6hkg\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.614553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614507 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.615012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.615012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.615012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.614639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69dc5200-1e97-473d-93a0-a02b6abe5654-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.716055 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716026 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-wtmp\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69dc5200-1e97-473d-93a0-a02b6abe5654-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-tls\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1197d210-3afa-4053-a758-ff0ce4f8e52f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716166 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716240 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f92v\" (UniqueName: \"kubernetes.io/projected/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-api-access-8f92v\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-metrics-client-ca\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hkg\" (UniqueName: \"kubernetes.io/projected/69dc5200-1e97-473d-93a0-a02b6abe5654-kube-api-access-w6hkg\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716324 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-textfile\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85fx\" (UniqueName: \"kubernetes.io/projected/052e5cc4-8625-46d3-b707-dcc9eec2728a-kube-api-access-s85fx\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716404 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-sys\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:27.716447 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 07 16:28:27.716577 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:27.716515 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls podName:69dc5200-1e97-473d-93a0-a02b6abe5654 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:28.216495408 +0000 UTC m=+312.525334669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-7glx6" (UID: "69dc5200-1e97-473d-93a0-a02b6abe5654") : secret "openshift-state-metrics-tls" not found Jul 07 16:28:27.717056 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:27.716754 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 16:28:27.717056 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:27.716792 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls podName:1197d210-3afa-4053-a758-ff0ce4f8e52f nodeName:}" failed. No retries permitted until 2026-07-07 16:28:28.216780042 +0000 UTC m=+312.525619301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-tlnrc" (UID: "1197d210-3afa-4053-a758-ff0ce4f8e52f") : secret "kube-state-metrics-tls" not found Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.717164 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.717440 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69dc5200-1e97-473d-93a0-a02b6abe5654-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.716448 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.717527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.717540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1197d210-3afa-4053-a758-ff0ce4f8e52f-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.717671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.717587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-root\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.718071 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.718006 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.718255 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.718231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.720933 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.720892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.722360 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.722336 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.751358 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.744474 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hkg\" (UniqueName: \"kubernetes.io/projected/69dc5200-1e97-473d-93a0-a02b6abe5654-kube-api-access-w6hkg\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:27.751358 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.747828 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f92v\" (UniqueName: \"kubernetes.io/projected/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-api-access-8f92v\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:27.818789 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-tls\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.818907 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818827 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.818907 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-metrics-client-ca\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.818907 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-textfile\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819058 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85fx\" (UniqueName: \"kubernetes.io/projected/052e5cc4-8625-46d3-b707-dcc9eec2728a-kube-api-access-s85fx\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819058 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818954 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-sys\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819058 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.818984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819058 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-root\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819246 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-wtmp\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819302 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819239 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-wtmp\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819305 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-sys\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819449 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/052e5cc4-8625-46d3-b707-dcc9eec2728a-root\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-metrics-client-ca\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-textfile\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.819859 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.819822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-accelerators-collector-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.822439 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.822403 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-tls\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.823729 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.823690 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/052e5cc4-8625-46d3-b707-dcc9eec2728a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.837190 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.837161 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85fx\" (UniqueName: \"kubernetes.io/projected/052e5cc4-8625-46d3-b707-dcc9eec2728a-kube-api-access-s85fx\") pod \"node-exporter-jtq8q\" (UID: \"052e5cc4-8625-46d3-b707-dcc9eec2728a\") " pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:27.857268 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.857237 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:27.891865 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:27.891669 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jtq8q" Jul 07 16:28:28.006305 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.006270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:28.157655 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.157525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:28.222849 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.222804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:28.223016 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.222936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:28.223087 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:28.223066 2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 16:28:28.223483 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:28.223151 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls podName:1197d210-3afa-4053-a758-ff0ce4f8e52f nodeName:}" failed. No retries permitted until 2026-07-07 16:28:29.223133704 +0000 UTC m=+313.531972967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-tlnrc" (UID: "1197d210-3afa-4053-a758-ff0ce4f8e52f") : secret "kube-state-metrics-tls" not found Jul 07 16:28:28.226768 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.226743 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/69dc5200-1e97-473d-93a0-a02b6abe5654-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7glx6\" (UID: \"69dc5200-1e97-473d-93a0-a02b6abe5654\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:28.373236 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.373201 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:28.438259 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.438006 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" Jul 07 16:28:28.544633 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.544492 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:28:28.549796 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.549774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.566698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566140 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 16:28:28.566698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566278 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 16:28:28.566698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566594 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 16:28:28.566938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566775 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:28.566938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566807 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 16:28:28.566938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.566926 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 16:28:28.567967 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.567126 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 16:28:28.567967 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.567179 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2plhw\"" Jul 07 16:28:28.567967 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.567293 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 16:28:28.567967 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.567426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 16:28:28.571524 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.571503 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 16:28:28.589563 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.589540 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:28:28.628657 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkz5\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.628970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.628992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.629015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.629039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.629082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.629137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.629388 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.629169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.729879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.729925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.729956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.730949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkz5\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.731020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.733222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.731404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.735638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.734778 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.735638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.734929 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.735638 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:28.735013 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 16:28:28.735638 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:28.735080 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls podName:a3cdca81-9ffb-49f2-8c31-46494b03a3b3 nodeName:}" failed. No retries permitted until 2026-07-07 16:28:29.235063129 +0000 UTC m=+313.543902388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3") : secret "alertmanager-main-tls" not found Jul 07 16:28:28.735638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.735348 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.738355 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.738326 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.738894 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.738714 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.738894 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.738819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.739350 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.739314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.739747 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.739711 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.740374 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.739964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.741504 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.741467 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.752637 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.752568 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkz5\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:28.807060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:28.807029 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:29.000378 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.000303 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:29.023864 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:29.023784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052e5cc4_8625_46d3_b707_dcc9eec2728a.slice/crio-5976e46a710cb5e184e57e26ba7dc08a3fd8b42eb81dacfec8956af43462216c WatchSource:0}: Error finding container 5976e46a710cb5e184e57e26ba7dc08a3fd8b42eb81dacfec8956af43462216c: Status 404 returned error can't find the container with id 5976e46a710cb5e184e57e26ba7dc08a3fd8b42eb81dacfec8956af43462216c Jul 07 16:28:29.188200 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.188172 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6"] Jul 07 16:28:29.190844 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:29.190807 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dc5200_1e97_473d_93a0_a02b6abe5654.slice/crio-3cf9627061b16b161285ad07a4436ac9ba6f7c7be2a13b4c539235e06e7d2d7a WatchSource:0}: Error finding container 3cf9627061b16b161285ad07a4436ac9ba6f7c7be2a13b4c539235e06e7d2d7a: Status 404 returned error can't find the container with id 3cf9627061b16b161285ad07a4436ac9ba6f7c7be2a13b4c539235e06e7d2d7a Jul 07 16:28:29.222323 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.222250 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mk524" event={"ID":"e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df","Type":"ContainerStarted","Data":"9b75598e66aa4449dd05d3ffef0998ef8f65c9b76bb8059fcfc1f2ad4aee3c85"} Jul 07 16:28:29.223702 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.223646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" event={"ID":"69dc5200-1e97-473d-93a0-a02b6abe5654","Type":"ContainerStarted","Data":"3cf9627061b16b161285ad07a4436ac9ba6f7c7be2a13b4c539235e06e7d2d7a"} Jul 07 16:28:29.225058 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.225031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtq8q" event={"ID":"052e5cc4-8625-46d3-b707-dcc9eec2728a","Type":"ContainerStarted","Data":"5976e46a710cb5e184e57e26ba7dc08a3fd8b42eb81dacfec8956af43462216c"} Jul 07 16:28:29.234730 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.234708 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:29.237226 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.237203 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1197d210-3afa-4053-a758-ff0ce4f8e52f-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tlnrc\" (UID: \"1197d210-3afa-4053-a758-ff0ce4f8e52f\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:29.238645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.238618 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:29.264441 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.264403 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mk524" podStartSLOduration=32.823090388 podStartE2EDuration="41.264388511s" totalImagesPullingTime="8.441298123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:29.262535213 +0000 UTC m=+313.571374493" watchObservedRunningTime="2026-07-07 16:28:29.264388511 +0000 UTC m=+313.573227808" Jul 07 16:28:29.335611 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.335570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:29.338123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.338099 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:29.352668 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.352647 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" Jul 07 16:28:29.416555 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.415818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:29.463894 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.463865 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:28:29.528931 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.528821 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc"] Jul 07 16:28:29.530978 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:29.530940 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1197d210_3afa_4053_a758_ff0ce4f8e52f.slice/crio-54834056add06709924740bea903aa0ede066305ba0e3e80a1a29b22123422ea WatchSource:0}: Error finding container 54834056add06709924740bea903aa0ede066305ba0e3e80a1a29b22123422ea: Status 404 returned error can't find the container with id 54834056add06709924740bea903aa0ede066305ba0e3e80a1a29b22123422ea Jul 07 16:28:29.663287 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.663255 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:28:29.665379 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:29.665348 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cdca81_9ffb_49f2_8c31_46494b03a3b3.slice/crio-1376257a04c8046d4a9f6e51b3ac7c01e0f1e62fbfe5d3f89c32567c32ec89d6 WatchSource:0}: Error finding container 1376257a04c8046d4a9f6e51b3ac7c01e0f1e62fbfe5d3f89c32567c32ec89d6: Status 404 returned error can't find the container with id 1376257a04c8046d4a9f6e51b3ac7c01e0f1e62fbfe5d3f89c32567c32ec89d6 Jul 07 16:28:29.726495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.726466 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:28:29.730062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.730038 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.734975 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.734945 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-66689c498-82snk\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.735101 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.734987 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-serving-cert\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" Jul 07 16:28:29.738322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738195 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-oauth-config\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" Jul 07 16:28:29.738322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738217 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"oauth-serving-cert\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" Jul 07 16:28:29.738322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738251 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-dockercfg-fg2nq\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-dockercfg-fg2nq\"" type="*v1.Secret" Jul 07 16:28:29.738322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738281 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"console-config\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" Jul 07 16:28:29.738322 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738292 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" Jul 07 16:28:29.738717 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:29.738363 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" Jul 07 16:28:29.788054 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.787989 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:28:29.839926 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.839894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840073 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.839938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840073 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.839968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840073 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.840022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.840113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.840144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tdr\" (UniqueName: \"kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.840237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.840177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941574 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941574 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tdr\" (UniqueName: \"kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.941791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.941754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:29.960184 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:29.960153 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tdr\" (UniqueName: \"kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:30.227387 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.227331 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:30.231232 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.231203 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"1376257a04c8046d4a9f6e51b3ac7c01e0f1e62fbfe5d3f89c32567c32ec89d6"} Jul 07 16:28:30.233134 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.233108 2569 generic.go:356] "Generic (PLEG): container finished" podID="052e5cc4-8625-46d3-b707-dcc9eec2728a" containerID="cfde5519e57961cf99fcf81bb9dcec6d31eb5ea17f90c09429109963112b948a" exitCode=0 Jul 07 16:28:30.233238 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.233196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtq8q" event={"ID":"052e5cc4-8625-46d3-b707-dcc9eec2728a","Type":"ContainerDied","Data":"cfde5519e57961cf99fcf81bb9dcec6d31eb5ea17f90c09429109963112b948a"} Jul 07 16:28:30.234651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.234600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" event={"ID":"1197d210-3afa-4053-a758-ff0ce4f8e52f","Type":"ContainerStarted","Data":"54834056add06709924740bea903aa0ede066305ba0e3e80a1a29b22123422ea"} Jul 07 16:28:30.236712 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.236691 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" event={"ID":"69dc5200-1e97-473d-93a0-a02b6abe5654","Type":"ContainerStarted","Data":"0fb7e38ff8bf9ef3a59c1e303400fccbca174622877396272cd271613e860e86"} Jul 07 16:28:30.236818 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.236741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" event={"ID":"69dc5200-1e97-473d-93a0-a02b6abe5654","Type":"ContainerStarted","Data":"9d896d6843b6bee97220d8777d3b06bc3056e73430b4619a65e2169c1dfd56ac"} Jul 07 16:28:30.631732 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.631688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:30.678310 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.678279 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fg2nq\"" Jul 07 16:28:30.729484 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.729447 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 16:28:30.732547 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.732487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:30.745414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.745388 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 16:28:30.752913 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.752885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:30.839204 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.839177 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:30.890478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.890412 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 16:28:30.894913 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:30.894888 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:30.942604 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942571 2569 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: failed to sync configmap cache: timed out waiting for the condition Jul 07 16:28:30.942604 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942594 2569 configmap.go:193] Couldn't get configMap openshift-console/service-ca: failed to sync configmap cache: timed out waiting for the condition Jul 07 16:28:30.942758 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942579 2569 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: failed to sync secret cache: timed out waiting for the condition Jul 07 16:28:30.942758 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942655 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert podName:d86990a7-4466-4352-b1d2-b0944d92e48c nodeName:}" failed. No retries permitted until 2026-07-07 16:28:31.442634892 +0000 UTC m=+315.751474155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert") pod "console-66689c498-82snk" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c") : failed to sync configmap cache: timed out waiting for the condition Jul 07 16:28:30.942758 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942680 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config podName:d86990a7-4466-4352-b1d2-b0944d92e48c nodeName:}" failed. No retries permitted until 2026-07-07 16:28:31.44266319 +0000 UTC m=+315.751502446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config") pod "console-66689c498-82snk" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c") : failed to sync secret cache: timed out waiting for the condition Jul 07 16:28:30.942758 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:28:30.942701 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca podName:d86990a7-4466-4352-b1d2-b0944d92e48c nodeName:}" failed. No retries permitted until 2026-07-07 16:28:31.442692149 +0000 UTC m=+315.751531406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca") pod "console-66689c498-82snk" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c") : failed to sync configmap cache: timed out waiting for the condition Jul 07 16:28:31.007355 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.007326 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:31.135802 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.135773 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 16:28:31.257297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.257223 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 16:28:31.268515 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.268489 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 16:28:31.295632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.295601 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:31.418085 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.417994 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:31.457266 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.457228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.457405 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.457281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.457405 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.457345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.457978 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.457914 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.458102 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.458084 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.460023 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.459999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") pod \"console-66689c498-82snk\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.560061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.559986 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:31.749516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:31.749486 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:32.022361 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:32.022320 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:32.177720 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:32.177689 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:32.323039 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:32.322945 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:34.463635 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:34.463606 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:28:34.489068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:34.489032 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:34.683574 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:34.683534 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:34.872732 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:34.872697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.072791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.072753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.287118 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.287034 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.395458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.395426 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.497735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.497700 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.867665 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.867613 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:35.965566 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:35.965529 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:37.782586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:37.782535 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:37.883422 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:37.883362 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:37.994763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:37.994735 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:28:38.028226 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:28:38.028080 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86990a7_4466_4352_b1d2_b0944d92e48c.slice/crio-632438791a7f77fe5f9a9337dee67749b11c11237f5aad06ac6416494d472268 WatchSource:0}: Error finding container 632438791a7f77fe5f9a9337dee67749b11c11237f5aad06ac6416494d472268: Status 404 returned error can't find the container with id 632438791a7f77fe5f9a9337dee67749b11c11237f5aad06ac6416494d472268 Jul 07 16:28:38.272276 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.272173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtq8q" event={"ID":"052e5cc4-8625-46d3-b707-dcc9eec2728a","Type":"ContainerStarted","Data":"d9bd6304308550bcb3d7fd9df070a2ccd1660df780ba23c0fdac0119b6e7a4d9"} Jul 07 16:28:38.272276 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.272233 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jtq8q" event={"ID":"052e5cc4-8625-46d3-b707-dcc9eec2728a","Type":"ContainerStarted","Data":"87088f54430ab07edf10fcd972b6b4667c2ed09015d68292a43a389b6e32a9bb"} Jul 07 16:28:38.274751 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.274708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-7h5hw" event={"ID":"819f7b33-44b8-4771-8329-e3216cc59113","Type":"ContainerStarted","Data":"3550000ca35c2f0a6529a63e3a80ddb3d80faa4a9af1c46b08f423522287cff2"} Jul 07 16:28:38.275259 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.274760 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:38.277932 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.277718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" event={"ID":"69dc5200-1e97-473d-93a0-a02b6abe5654","Type":"ContainerStarted","Data":"e754dda86394a59891559f7b56f9865b0ea613165dba5c877234bae76cfbfff2"} Jul 07 16:28:38.280022 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.279879 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0" exitCode=0 Jul 07 16:28:38.280022 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.279957 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0"} Jul 07 16:28:38.282063 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.282025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66689c498-82snk" event={"ID":"d86990a7-4466-4352-b1d2-b0944d92e48c","Type":"ContainerStarted","Data":"632438791a7f77fe5f9a9337dee67749b11c11237f5aad06ac6416494d472268"} Jul 07 16:28:38.287163 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.287146 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-7h5hw" Jul 07 16:28:38.310415 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.310363 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jtq8q" podStartSLOduration=10.273545572 podStartE2EDuration="11.310345358s" totalImagesPullingTime="962.105592ms" totalInitContainerRuntime="74.694194ms" isStatefulPod=true podCreationTimestamp="2026-07-07 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:38.309703877 +0000 UTC m=+322.618543156" watchObservedRunningTime="2026-07-07 16:28:38.310345358 +0000 UTC m=+322.619184637" Jul 07 16:28:38.401521 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.400395 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-7h5hw" podStartSLOduration=1.7299235990000001 podStartE2EDuration="21.4003748s" totalImagesPullingTime="19.670451201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:38.397507856 +0000 UTC m=+322.706347135" watchObservedRunningTime="2026-07-07 16:28:38.4003748 +0000 UTC m=+322.709214080" Jul 07 16:28:38.439379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:38.439319 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7glx6" podStartSLOduration=2.924387196 podStartE2EDuration="11.439299824s" totalImagesPullingTime="8.514912628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:38.438506488 +0000 UTC m=+322.747345767" watchObservedRunningTime="2026-07-07 16:28:38.439299824 +0000 UTC m=+322.748139102" Jul 07 16:28:39.297334 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:39.297234 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" event={"ID":"1197d210-3afa-4053-a758-ff0ce4f8e52f","Type":"ContainerStarted","Data":"8b08eed55c90168a54f55bada012eeefdf3abbf5218c70c72b1a776b756eabf1"} Jul 07 16:28:39.298037 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:39.297683 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" event={"ID":"1197d210-3afa-4053-a758-ff0ce4f8e52f","Type":"ContainerStarted","Data":"fed178d80e6fcb38aaaef9bc31a6442b793df705111769434d9cfaeaf6526e83"} Jul 07 16:28:39.298037 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:39.297740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" event={"ID":"1197d210-3afa-4053-a758-ff0ce4f8e52f","Type":"ContainerStarted","Data":"bf5b201dec81c56d395e3c909aec0bb0d3100cee9f521b9eff292a28642d86c7"} Jul 07 16:28:39.320144 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:39.319435 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tlnrc" podStartSLOduration=3.192377081 podStartE2EDuration="12.319413548s" totalImagesPullingTime="9.127036467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:39.317000386 +0000 UTC m=+323.625839665" watchObservedRunningTime="2026-07-07 16:28:39.319413548 +0000 UTC m=+323.628252833" Jul 07 16:28:40.482000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:40.481222 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:28:40.565609 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:40.565541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:40.667627 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:40.667585 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:40.770102 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:40.770014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:43.319351 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.319307 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758"} Jul 07 16:28:43.319897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.319433 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1"} Jul 07 16:28:43.319897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.319475 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13"} Jul 07 16:28:43.319897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.319518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9"} Jul 07 16:28:43.319897 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.319567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b"} Jul 07 16:28:43.321208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.321173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66689c498-82snk" event={"ID":"d86990a7-4466-4352-b1d2-b0944d92e48c","Type":"ContainerStarted","Data":"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53"} Jul 07 16:28:43.341262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:43.341206 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66689c498-82snk" podStartSLOduration=9.929577794 podStartE2EDuration="14.341187366s" totalImagesPullingTime="4.411609572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:43.339180701 +0000 UTC m=+327.648019981" watchObservedRunningTime="2026-07-07 16:28:43.341187366 +0000 UTC m=+327.650026645" Jul 07 16:28:45.334905 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:45.334872 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerStarted","Data":"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4"} Jul 07 16:28:45.370186 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:45.370131 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.276098203 podStartE2EDuration="17.370109037s" totalImagesPullingTime="14.060293928s" totalInitContainerRuntime="33.716906ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:28:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:28:45.368198355 +0000 UTC m=+329.677037633" watchObservedRunningTime="2026-07-07 16:28:45.370109037 +0000 UTC m=+329.678948316" Jul 07 16:28:45.493713 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:45.493681 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:45.565578 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:45.565548 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:46.141415 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:46.141378 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:46.416490 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:46.416417 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:49.854529 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:49.854496 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:49.956062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:49.956036 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:28:51.560624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:51.560595 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66689c498-82snk" Jul 07 16:28:57.380909 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:57.380828 2569 generic.go:356] "Generic (PLEG): container finished" podID="9b9be938-cda2-46fb-bab1-87b3b8584302" containerID="b2dedacb8e64ca5d6c3b9e31c19bc7aeb1f905796370ee8e9c5592257201bf72" exitCode=0 Jul 07 16:28:57.381226 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:57.380900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" event={"ID":"9b9be938-cda2-46fb-bab1-87b3b8584302","Type":"ContainerDied","Data":"b2dedacb8e64ca5d6c3b9e31c19bc7aeb1f905796370ee8e9c5592257201bf72"} Jul 07 16:28:57.381226 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:57.381219 2569 scope.go:117] "RemoveContainer" containerID="b2dedacb8e64ca5d6c3b9e31c19bc7aeb1f905796370ee8e9c5592257201bf72" Jul 07 16:28:58.387454 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:28:58.387419 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-zmhwf" event={"ID":"9b9be938-cda2-46fb-bab1-87b3b8584302","Type":"ContainerStarted","Data":"c0305b7ba0d634c393c9feb48f6f3e5937fc3453c3cf5eb249eb5001183e22de"} Jul 07 16:29:05.511053 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.511012 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" podUID="2f155471-7d49-4047-bcba-9647f0016638" containerName="registry" containerID="cri-o://8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02" gracePeriod=30 Jul 07 16:29:05.756548 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.756527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:29:05.841548 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841526 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841568 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7h4l\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841619 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841865 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841712 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841865 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841794 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.841865 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841821 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.842044 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.841877 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") pod \"2f155471-7d49-4047-bcba-9647f0016638\" (UID: \"2f155471-7d49-4047-bcba-9647f0016638\") " Jul 07 16:29:05.842098 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.842064 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:05.842432 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.842148 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:05.842432 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.842216 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-registry-certificates\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.844178 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.844151 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:05.844287 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.844183 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l" (OuterVolumeSpecName: "kube-api-access-v7h4l") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "kube-api-access-v7h4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:05.844287 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.844259 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:05.844440 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.844421 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:05.844503 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.844454 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:05.850951 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.850928 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2f155471-7d49-4047-bcba-9647f0016638" (UID: "2f155471-7d49-4047-bcba-9647f0016638"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:29:05.942807 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942774 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f155471-7d49-4047-bcba-9647f0016638-ca-trust-extracted\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942807 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942804 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7h4l\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-kube-api-access-v7h4l\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942818 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-installation-pull-secrets\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942830 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f155471-7d49-4047-bcba-9647f0016638-trusted-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942869 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f155471-7d49-4047-bcba-9647f0016638-image-registry-private-configuration\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942882 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-bound-sa-token\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:05.942969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:05.942894 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f155471-7d49-4047-bcba-9647f0016638-registry-tls\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:06.420883 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.420855 2569 generic.go:356] "Generic (PLEG): container finished" podID="2f155471-7d49-4047-bcba-9647f0016638" containerID="8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02" exitCode=0 Jul 07 16:29:06.420982 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.420891 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" Jul 07 16:29:06.420982 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.420919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" event={"ID":"2f155471-7d49-4047-bcba-9647f0016638","Type":"ContainerDied","Data":"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02"} Jul 07 16:29:06.420982 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.420966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85dfbb85f8-tcpsz" event={"ID":"2f155471-7d49-4047-bcba-9647f0016638","Type":"ContainerDied","Data":"5335ccca0b728301d088d5d072c83830305542fa44ea1d3214d0da1650be44de"} Jul 07 16:29:06.421099 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.420990 2569 scope.go:117] "RemoveContainer" containerID="8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02" Jul 07 16:29:06.429218 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.429187 2569 scope.go:117] "RemoveContainer" containerID="8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02" Jul 07 16:29:06.429471 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:06.429452 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02\": container with ID starting with 8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02 not found: ID does not exist" containerID="8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02" Jul 07 16:29:06.429530 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.429479 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02"} err="failed to get container status \"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02\": rpc error: code = NotFound desc = could not find container \"8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02\": container with ID starting with 8ddfeb9b422db526d1876734181ab01206f129f01594632ccbb4a41711c96b02 not found: ID does not exist" Jul 07 16:29:06.441232 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.441210 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:29:06.444774 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:06.444753 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85dfbb85f8-tcpsz"] Jul 07 16:29:08.266973 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.266898 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f155471-7d49-4047-bcba-9647f0016638" path="/var/lib/kubelet/pods/2f155471-7d49-4047-bcba-9647f0016638/volumes" Jul 07 16:29:08.345060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.345003 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-66689c498-82snk" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" containerName="console" containerID="cri-o://c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53" gracePeriod=15 Jul 07 16:29:08.635951 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.635914 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66689c498-82snk" Jul 07 16:29:08.763948 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.763916 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.763971 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.763995 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764037 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764069 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764123 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764124 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9tdr\" (UniqueName: \"kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr\") pod \"d86990a7-4466-4352-b1d2-b0944d92e48c\" (UID: \"d86990a7-4466-4352-b1d2-b0944d92e48c\") " Jul 07 16:29:08.764389 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764323 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:08.764489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764394 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca" (OuterVolumeSpecName: "service-ca") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:08.764489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764411 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:08.764489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764453 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-trusted-ca-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.764631 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.764612 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config" (OuterVolumeSpecName: "console-config") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:08.766541 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.766513 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr" (OuterVolumeSpecName: "kube-api-access-q9tdr") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "kube-api-access-q9tdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:08.766715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.766686 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:08.766819 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.766735 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d86990a7-4466-4352-b1d2-b0944d92e48c" (UID: "d86990a7-4466-4352-b1d2-b0944d92e48c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:08.865280 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865250 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.865280 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865278 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-oauth-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.865468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865292 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.865468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865305 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d86990a7-4466-4352-b1d2-b0944d92e48c-console-oauth-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.865468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865318 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d86990a7-4466-4352-b1d2-b0944d92e48c-console-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:08.865468 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:08.865330 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tdr\" (UniqueName: \"kubernetes.io/projected/d86990a7-4466-4352-b1d2-b0944d92e48c-kube-api-access-q9tdr\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:09.435358 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.435326 2569 generic.go:356] "Generic (PLEG): container finished" podID="d86990a7-4466-4352-b1d2-b0944d92e48c" containerID="c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53" exitCode=0 Jul 07 16:29:09.435793 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.435372 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66689c498-82snk" Jul 07 16:29:09.435793 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.435379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66689c498-82snk" event={"ID":"d86990a7-4466-4352-b1d2-b0944d92e48c","Type":"ContainerDied","Data":"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53"} Jul 07 16:29:09.435793 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.435421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66689c498-82snk" event={"ID":"d86990a7-4466-4352-b1d2-b0944d92e48c","Type":"ContainerDied","Data":"632438791a7f77fe5f9a9337dee67749b11c11237f5aad06ac6416494d472268"} Jul 07 16:29:09.435793 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.435442 2569 scope.go:117] "RemoveContainer" containerID="c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53" Jul 07 16:29:09.444853 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.444806 2569 scope.go:117] "RemoveContainer" containerID="c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53" Jul 07 16:29:09.445126 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:09.445104 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53\": container with ID starting with c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53 not found: ID does not exist" containerID="c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53" Jul 07 16:29:09.445210 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.445137 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53"} err="failed to get container status \"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53\": rpc error: code = NotFound desc = could not find container \"c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53\": container with ID starting with c2610030aab3fb52ad923940e63edd58c64179d6d5ff2d925399581fd2a2fc53 not found: ID does not exist" Jul 07 16:29:09.458518 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.458497 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:29:09.461542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:09.461521 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66689c498-82snk"] Jul 07 16:29:10.266104 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:10.266072 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" path="/var/lib/kubelet/pods/d86990a7-4466-4352-b1d2-b0944d92e48c/volumes" Jul 07 16:29:12.453203 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:12.453165 2569 generic.go:356] "Generic (PLEG): container finished" podID="a1983aed-25a3-4fb3-b9cc-08ebdbbd7691" containerID="ce0c35356ab1a39e27c260ee05041f6443a01a5df58165adbefa9ff4699f1250" exitCode=0 Jul 07 16:29:12.453564 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:12.453240 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" event={"ID":"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691","Type":"ContainerDied","Data":"ce0c35356ab1a39e27c260ee05041f6443a01a5df58165adbefa9ff4699f1250"} Jul 07 16:29:12.453564 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:12.453544 2569 scope.go:117] "RemoveContainer" containerID="ce0c35356ab1a39e27c260ee05041f6443a01a5df58165adbefa9ff4699f1250" Jul 07 16:29:13.459247 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:13.459213 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-v7n9v" event={"ID":"a1983aed-25a3-4fb3-b9cc-08ebdbbd7691","Type":"ContainerStarted","Data":"4933efb3c6bd5dccd68a910647f7194619e3a1aa19c8ed517cfd0116cd17cd53"} Jul 07 16:29:20.079704 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:20.079663 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:20.153739 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:20.153713 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:30.127229 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.127191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:29:30.129604 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.129572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d189f61-09ee-44d8-a2e0-38b22a416ad4-metrics-certs\") pod \"network-metrics-daemon-cvcbn\" (UID: \"0d189f61-09ee-44d8-a2e0-38b22a416ad4\") " pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:29:30.265411 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.265382 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sncwv\"" Jul 07 16:29:30.272860 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.272844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cvcbn" Jul 07 16:29:30.400548 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.400479 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cvcbn"] Jul 07 16:29:30.402553 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:29:30.402518 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d189f61_09ee_44d8_a2e0_38b22a416ad4.slice/crio-09cacaa22190070597fb8556190f9cbea5b369239730159a330a378544c818a9 WatchSource:0}: Error finding container 09cacaa22190070597fb8556190f9cbea5b369239730159a330a378544c818a9: Status 404 returned error can't find the container with id 09cacaa22190070597fb8556190f9cbea5b369239730159a330a378544c818a9 Jul 07 16:29:30.523111 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:30.523077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvcbn" event={"ID":"0d189f61-09ee-44d8-a2e0-38b22a416ad4","Type":"ContainerStarted","Data":"09cacaa22190070597fb8556190f9cbea5b369239730159a330a378544c818a9"} Jul 07 16:29:31.528765 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:31.528731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvcbn" event={"ID":"0d189f61-09ee-44d8-a2e0-38b22a416ad4","Type":"ContainerStarted","Data":"fb5d1946f53679a1cba4e74847240b4a2e2fd6ecc45f56f8f50f683d556bacb8"} Jul 07 16:29:31.529068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:31.528780 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cvcbn" event={"ID":"0d189f61-09ee-44d8-a2e0-38b22a416ad4","Type":"ContainerStarted","Data":"9f39007c54f22d246905b5e3b4d40c417921b28cca7e92c4a68af60daed284d4"} Jul 07 16:29:31.701270 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:31.701214 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:31.780854 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:31.780814 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:33.900856 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:33.900802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:34.008306 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:34.008275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:38.454757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:38.454720 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:38.554339 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:38.554309 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:47.742300 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:47.742269 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:47.843305 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:47.843274 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:48.776353 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.776297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cvcbn" podStartSLOduration=390.846044257 podStartE2EDuration="6m31.776279109s" totalImagesPullingTime="930.234852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:29:31.558179485 +0000 UTC m=+375.867018764" watchObservedRunningTime="2026-07-07 16:29:48.776279109 +0000 UTC m=+393.085118386" Jul 07 16:29:48.777990 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.777962 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:48.778317 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778291 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="alertmanager" containerID="cri-o://77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b" gracePeriod=120 Jul 07 16:29:48.778381 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778348 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-metric" containerID="cri-o://f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758" gracePeriod=120 Jul 07 16:29:48.778429 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778381 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="prom-label-proxy" containerID="cri-o://ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4" gracePeriod=120 Jul 07 16:29:48.778479 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778356 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-web" containerID="cri-o://2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13" gracePeriod=120 Jul 07 16:29:48.778531 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778370 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="config-reloader" containerID="cri-o://266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9" gracePeriod=120 Jul 07 16:29:48.778582 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:48.778394 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy" containerID="cri-o://1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1" gracePeriod=120 Jul 07 16:29:49.605400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605363 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4" exitCode=0 Jul 07 16:29:49.605400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605390 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758" exitCode=0 Jul 07 16:29:49.605400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605399 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1" exitCode=0 Jul 07 16:29:49.605400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605405 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9" exitCode=0 Jul 07 16:29:49.605400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605410 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b" exitCode=0 Jul 07 16:29:49.605686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4"} Jul 07 16:29:49.605686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758"} Jul 07 16:29:49.605686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605555 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1"} Jul 07 16:29:49.605686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9"} Jul 07 16:29:49.605686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:49.605604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b"} Jul 07 16:29:50.025248 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.025219 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.172546 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172465 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.172546 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172501 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.172546 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172530 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.172816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172568 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.172816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172598 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.172816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172623 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqkz5\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173015 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172907 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:50.173076 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.172647 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173076 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173052 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173086 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173125 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173158 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173308 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173195 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173308 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173222 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca\") pod \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\" (UID: \"a3cdca81-9ffb-49f2-8c31-46494b03a3b3\") " Jul 07 16:29:50.173701 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173459 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.173863 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.173815 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:29:50.174039 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.174010 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:29:50.175595 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.175542 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume" (OuterVolumeSpecName: "config-volume") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.175710 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.175680 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.176133 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.176094 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.176386 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.176362 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out" (OuterVolumeSpecName: "config-out") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:29:50.176575 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.176550 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.176866 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.176819 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:50.177009 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.176989 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5" (OuterVolumeSpecName: "kube-api-access-cqkz5") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "kube-api-access-cqkz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:29:50.177541 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.177514 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.180414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.180356 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.185966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.185944 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config" (OuterVolumeSpecName: "web-config") pod "a3cdca81-9ffb-49f2-8c31-46494b03a3b3" (UID: "a3cdca81-9ffb-49f2-8c31-46494b03a3b3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:29:50.273844 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273813 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-main-tls\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273851 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-volume\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273864 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkz5\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-kube-api-access-cqkz5\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273873 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273882 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-cluster-tls-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273890 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-tls-assets\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273898 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273913 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-config-out\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.273925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273922 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-alertmanager-main-db\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.274157 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273930 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-metrics-client-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.274157 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273939 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-web-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.274157 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.273947 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3cdca81-9ffb-49f2-8c31-46494b03a3b3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:29:50.612132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.612099 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerID="2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13" exitCode=0 Jul 07 16:29:50.612275 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.612186 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.612275 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.612187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13"} Jul 07 16:29:50.612392 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.612280 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3cdca81-9ffb-49f2-8c31-46494b03a3b3","Type":"ContainerDied","Data":"1376257a04c8046d4a9f6e51b3ac7c01e0f1e62fbfe5d3f89c32567c32ec89d6"} Jul 07 16:29:50.612392 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.612334 2569 scope.go:117] "RemoveContainer" containerID="ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4" Jul 07 16:29:50.619809 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.619793 2569 scope.go:117] "RemoveContainer" containerID="f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758" Jul 07 16:29:50.628472 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.628453 2569 scope.go:117] "RemoveContainer" containerID="1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1" Jul 07 16:29:50.634846 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.634811 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:50.636795 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.636780 2569 scope.go:117] "RemoveContainer" containerID="2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13" Jul 07 16:29:50.638340 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.638323 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:50.643330 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.643308 2569 scope.go:117] "RemoveContainer" containerID="266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9" Jul 07 16:29:50.649908 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.649890 2569 scope.go:117] "RemoveContainer" containerID="77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b" Jul 07 16:29:50.656193 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.656172 2569 scope.go:117] "RemoveContainer" containerID="117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0" Jul 07 16:29:50.662924 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.662910 2569 scope.go:117] "RemoveContainer" containerID="ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4" Jul 07 16:29:50.663162 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.663145 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4\": container with ID starting with ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4 not found: ID does not exist" containerID="ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4" Jul 07 16:29:50.663225 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663173 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4"} err="failed to get container status \"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4\": rpc error: code = NotFound desc = could not find container \"ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4\": container with ID starting with ce0226c1350a56a6e2e1fb566e1f3b4fba3575ac136ed5c4ede3b0638f241dd4 not found: ID does not exist" Jul 07 16:29:50.663225 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663198 2569 scope.go:117] "RemoveContainer" containerID="f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758" Jul 07 16:29:50.663455 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.663440 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758\": container with ID starting with f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758 not found: ID does not exist" containerID="f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758" Jul 07 16:29:50.663497 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663461 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758"} err="failed to get container status \"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758\": rpc error: code = NotFound desc = could not find container \"f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758\": container with ID starting with f68d6915007245404ce578b7ce52f7ed3772bc2f7f08697aa9791ffb255b3758 not found: ID does not exist" Jul 07 16:29:50.663497 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663483 2569 scope.go:117] "RemoveContainer" containerID="1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1" Jul 07 16:29:50.663706 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.663689 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1\": container with ID starting with 1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1 not found: ID does not exist" containerID="1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1" Jul 07 16:29:50.663749 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663711 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1"} err="failed to get container status \"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1\": rpc error: code = NotFound desc = could not find container \"1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1\": container with ID starting with 1adc57a8b4ae458cac6736099b171289e229a41fc302d67cc5063605e42271a1 not found: ID does not exist" Jul 07 16:29:50.663749 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663725 2569 scope.go:117] "RemoveContainer" containerID="2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13" Jul 07 16:29:50.663941 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.663925 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13\": container with ID starting with 2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13 not found: ID does not exist" containerID="2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13" Jul 07 16:29:50.663986 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663945 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13"} err="failed to get container status \"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13\": rpc error: code = NotFound desc = could not find container \"2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13\": container with ID starting with 2f1815bad7a127a5264a3aab74ec64d0ff7fcd5948ef2db9246085da92a81e13 not found: ID does not exist" Jul 07 16:29:50.663986 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.663957 2569 scope.go:117] "RemoveContainer" containerID="266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9" Jul 07 16:29:50.664172 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.664155 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9\": container with ID starting with 266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9 not found: ID does not exist" containerID="266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9" Jul 07 16:29:50.664239 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.664189 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9"} err="failed to get container status \"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9\": rpc error: code = NotFound desc = could not find container \"266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9\": container with ID starting with 266ed6076419cccea73258a5b701a94f70794a6b9c6ca50600b70272bcd77ea9 not found: ID does not exist" Jul 07 16:29:50.664239 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.664209 2569 scope.go:117] "RemoveContainer" containerID="77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b" Jul 07 16:29:50.664424 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.664408 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b\": container with ID starting with 77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b not found: ID does not exist" containerID="77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b" Jul 07 16:29:50.664481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.664429 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b"} err="failed to get container status \"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b\": rpc error: code = NotFound desc = could not find container \"77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b\": container with ID starting with 77b6790a51fb599fc2f535c24aa027da8c4e86202d83155b4de83d3321bd8e8b not found: ID does not exist" Jul 07 16:29:50.664481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.664447 2569 scope.go:117] "RemoveContainer" containerID="117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0" Jul 07 16:29:50.664660 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:29:50.664643 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0\": container with ID starting with 117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0 not found: ID does not exist" containerID="117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0" Jul 07 16:29:50.664700 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.664665 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0"} err="failed to get container status \"117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0\": rpc error: code = NotFound desc = could not find container \"117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0\": container with ID starting with 117793a90c51077acd9b892d12732cb5ccb13becff7a865f928f506803987fa0 not found: ID does not exist" Jul 07 16:29:50.666688 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666666 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:50.666962 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666949 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="init-config-reloader" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666965 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="init-config-reloader" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666974 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666980 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666987 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="prom-label-proxy" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666992 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="prom-label-proxy" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.666998 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-web" Jul 07 16:29:50.667013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667003 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-web" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667016 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f155471-7d49-4047-bcba-9647f0016638" containerName="registry" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667022 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f155471-7d49-4047-bcba-9647f0016638" containerName="registry" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667030 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="alertmanager" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667035 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="alertmanager" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667044 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="config-reloader" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667052 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="config-reloader" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667060 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-metric" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667068 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-metric" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667077 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" containerName="console" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667082 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" containerName="console" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667133 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-metric" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667141 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="prom-label-proxy" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667149 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="config-reloader" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667154 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86990a7-4466-4352-b1d2-b0944d92e48c" containerName="console" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667160 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667167 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="alertmanager" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667173 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" containerName="kube-rbac-proxy-web" Jul 07 16:29:50.667252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.667178 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f155471-7d49-4047-bcba-9647f0016638" containerName="registry" Jul 07 16:29:50.669499 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.669482 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.671938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.671913 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 16:29:50.671938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.671932 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 16:29:50.672091 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 16:29:50.672157 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 16:29:50.672212 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 16:29:50.672384 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672362 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 16:29:50.672477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 16:29:50.672542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672504 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2plhw\"" Jul 07 16:29:50.672647 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.672552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 16:29:50.678816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.678798 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 16:29:50.686288 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.686266 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:50.777042 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777012 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-out\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777125 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777187 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777187 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777291 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cpn\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-kube-api-access-42cpn\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777291 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777395 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777395 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777393 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777495 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-web-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.777641 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.777501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cpn\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-kube-api-access-42cpn\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878414 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-web-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-out\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.878773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.878729 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.879630 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.879329 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.879737 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.879669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bcb872a-4a05-4343-b439-c14f71d5e2b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.881586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.881487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.881586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.881580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.881729 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.881584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.881783 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.881770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.882127 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.882103 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-out\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.882219 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.882188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.882293 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.882274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-web-config\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.882333 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.882317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.883291 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.883274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5bcb872a-4a05-4343-b439-c14f71d5e2b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.887741 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.887725 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cpn\" (UniqueName: \"kubernetes.io/projected/5bcb872a-4a05-4343-b439-c14f71d5e2b1-kube-api-access-42cpn\") pod \"alertmanager-main-0\" (UID: \"5bcb872a-4a05-4343-b439-c14f71d5e2b1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:50.979394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:50.979371 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 16:29:51.117367 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:51.117337 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 16:29:51.120486 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:29:51.120461 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcb872a_4a05_4343_b439_c14f71d5e2b1.slice/crio-c668516e529cdbbce0e105ae7e06aa6e4627a55677ab6c04e29db98294e8df85 WatchSource:0}: Error finding container c668516e529cdbbce0e105ae7e06aa6e4627a55677ab6c04e29db98294e8df85: Status 404 returned error can't find the container with id c668516e529cdbbce0e105ae7e06aa6e4627a55677ab6c04e29db98294e8df85 Jul 07 16:29:51.618781 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:51.618743 2569 generic.go:356] "Generic (PLEG): container finished" podID="5bcb872a-4a05-4343-b439-c14f71d5e2b1" containerID="d9c27d3f0ce24517220c6a7506d245870ad038db5864862a0929c5623f22fa24" exitCode=0 Jul 07 16:29:51.618991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:51.618856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerDied","Data":"d9c27d3f0ce24517220c6a7506d245870ad038db5864862a0929c5623f22fa24"} Jul 07 16:29:51.618991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:51.618923 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"c668516e529cdbbce0e105ae7e06aa6e4627a55677ab6c04e29db98294e8df85"} Jul 07 16:29:52.266308 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.266244 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3cdca81-9ffb-49f2-8c31-46494b03a3b3" path="/var/lib/kubelet/pods/a3cdca81-9ffb-49f2-8c31-46494b03a3b3/volumes" Jul 07 16:29:52.627597 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"86ee3162220d01bff76c0b243b325831b0a34bf4d6f6ee7b64d91c383654907b"} Jul 07 16:29:52.627722 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"80e18d6b3ebbfd5c6f015d4db35e85611f7b89030fcf7e8df2111f3d23ebbf74"} Jul 07 16:29:52.627722 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"98028afd1d5f54977e0412961d4ebc4e0e62f3a822d2869bcf808e973d2f568b"} Jul 07 16:29:52.627722 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627682 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"54f7e2f29a06a23c3a4e99ee1dbed1de8b0f075efe830a083ed6f9c485ccc82b"} Jul 07 16:29:52.627722 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627705 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"ec29d354be8f5a5477142033ed278929c830befe53944fdc122ded8bda23e88c"} Jul 07 16:29:52.627896 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.627728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5bcb872a-4a05-4343-b439-c14f71d5e2b1","Type":"ContainerStarted","Data":"49906b7fc2a4c2eac513fde469722d32d7ed6ae50147740bc4f28cf52a2f7778"} Jul 07 16:29:52.659137 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.659091 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.645548356 podStartE2EDuration="2.659075739s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.527383ms" isStatefulPod=false podCreationTimestamp="2026-07-07 16:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:29:52.656733746 +0000 UTC m=+396.965573027" watchObservedRunningTime="2026-07-07 16:29:52.659075739 +0000 UTC m=+396.967915016" Jul 07 16:29:52.864965 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.864319 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-66dfb77967-cz7j7"] Jul 07 16:29:52.867306 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.867284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.870478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.870456 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 16:29:52.870624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.870601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 16:29:52.870928 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.870783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 16:29:52.870928 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.870817 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 16:29:52.871181 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.871080 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cv6mx\"" Jul 07 16:29:52.871571 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.871543 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 16:29:52.876425 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.876406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 16:29:52.888591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.888541 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66dfb77967-cz7j7"] Jul 07 16:29:52.995745 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-serving-certs-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995865 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-metrics-client-ca\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.995974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzhp\" (UniqueName: \"kubernetes.io/projected/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-kube-api-access-8hzhp\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:52.996067 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:52.995989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-federate-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096332 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzhp\" (UniqueName: \"kubernetes.io/projected/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-kube-api-access-8hzhp\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096407 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-federate-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096407 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096507 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096485 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096544 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-serving-certs-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096598 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-metrics-client-ca\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096637 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.096687 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.096653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.097348 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.097326 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-serving-certs-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.097601 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.097580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-metrics-client-ca\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.097650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.097586 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-trusted-ca-bundle\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.099012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.098985 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.099116 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.099097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-federate-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.099293 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.099278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-secret-telemeter-client\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.099467 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.099447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-telemeter-client-tls\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.105956 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.105937 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzhp\" (UniqueName: \"kubernetes.io/projected/05c7f4a6-4bf9-4fd8-beb5-24943180e7ea-kube-api-access-8hzhp\") pod \"telemeter-client-66dfb77967-cz7j7\" (UID: \"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea\") " pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.177054 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.177008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" Jul 07 16:29:53.325120 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.325097 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-66dfb77967-cz7j7"] Jul 07 16:29:53.327088 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:29:53.327061 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c7f4a6_4bf9_4fd8_beb5_24943180e7ea.slice/crio-2727ffeff86d5d8217dd7fa2987f777242a7d718f736f322c1113a8dc1df21dc WatchSource:0}: Error finding container 2727ffeff86d5d8217dd7fa2987f777242a7d718f736f322c1113a8dc1df21dc: Status 404 returned error can't find the container with id 2727ffeff86d5d8217dd7fa2987f777242a7d718f736f322c1113a8dc1df21dc Jul 07 16:29:53.633128 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:53.633100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" event={"ID":"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea","Type":"ContainerStarted","Data":"2727ffeff86d5d8217dd7fa2987f777242a7d718f736f322c1113a8dc1df21dc"} Jul 07 16:29:55.643083 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:55.643056 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" event={"ID":"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea","Type":"ContainerStarted","Data":"0dcbca3f21fb93e5d5369b71dbb88a3132e0d2102cf0d540e5ee547e86332778"} Jul 07 16:29:55.643384 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:55.643122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" event={"ID":"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea","Type":"ContainerStarted","Data":"76b548189fa090bf0fd89de3e9895dab8e23a35b64ed4bbfc860c98a50677f80"} Jul 07 16:29:56.648814 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:56.648776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" event={"ID":"05c7f4a6-4bf9-4fd8-beb5-24943180e7ea","Type":"ContainerStarted","Data":"8dd8d49f1ce91df09daed9f7d80b12fc04f3de566cf806cf0a40b5e0fdf2bd8a"} Jul 07 16:29:56.679433 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:56.679367 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-66dfb77967-cz7j7" podStartSLOduration=2.540897368 podStartE2EDuration="4.679347407s" totalImagesPullingTime="2.138450039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:29:56.678933905 +0000 UTC m=+400.987773182" watchObservedRunningTime="2026-07-07 16:29:56.679347407 +0000 UTC m=+400.988186687" Jul 07 16:29:57.309409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.309383 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:29:57.311905 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.311888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.316474 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.316449 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 16:29:57.316590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.316571 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 16:29:57.316715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.316695 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 16:29:57.317776 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.317756 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 16:29:57.317890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.317871 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fg2nq\"" Jul 07 16:29:57.317957 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.317904 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 16:29:57.325641 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.325611 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 16:29:57.329636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.329615 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:29:57.382281 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.382253 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:57.431339 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpkz\" (UniqueName: \"kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431451 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431451 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431451 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.431682 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.431617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.443957 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.443936 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:29:57.532282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532363 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532363 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpkz\" (UniqueName: \"kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532363 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532467 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532467 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.532467 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.532451 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.533111 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.533091 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.533190 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.533093 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.533190 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.533134 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.533401 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.533377 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.534856 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.534815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.534984 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.534963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.541763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.541738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpkz\" (UniqueName: \"kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz\") pod \"console-67bf77fd7-pdnwr\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.621666 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.621642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:29:57.748775 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:57.748742 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:29:57.751357 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:29:57.751327 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2aa092_6044_42ad_bad2_3cc5a2055bde.slice/crio-081318b9221b2c4b0ffc14a0e867c407c783e61aa6ca7d3408297e1f54e2a974 WatchSource:0}: Error finding container 081318b9221b2c4b0ffc14a0e867c407c783e61aa6ca7d3408297e1f54e2a974: Status 404 returned error can't find the container with id 081318b9221b2c4b0ffc14a0e867c407c783e61aa6ca7d3408297e1f54e2a974 Jul 07 16:29:58.659245 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:58.659205 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67bf77fd7-pdnwr" event={"ID":"ea2aa092-6044-42ad-bad2-3cc5a2055bde","Type":"ContainerStarted","Data":"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4"} Jul 07 16:29:58.659423 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:58.659263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67bf77fd7-pdnwr" event={"ID":"ea2aa092-6044-42ad-bad2-3cc5a2055bde","Type":"ContainerStarted","Data":"081318b9221b2c4b0ffc14a0e867c407c783e61aa6ca7d3408297e1f54e2a974"} Jul 07 16:29:58.681235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:29:58.681169 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67bf77fd7-pdnwr" podStartSLOduration=1.68114666 podStartE2EDuration="1.68114666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:29:58.680709675 +0000 UTC m=+402.989548954" watchObservedRunningTime="2026-07-07 16:29:58.68114666 +0000 UTC m=+402.989985941" Jul 07 16:30:01.085447 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:30:01.085399 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" podUID="3af45c32-632e-4dbb-8f65-9a30f443b58f" Jul 07 16:30:01.672325 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:01.672290 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:30:02.975223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:02.975191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:30:02.977787 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:02.977757 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3af45c32-632e-4dbb-8f65-9a30f443b58f-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5qcwh\" (UID: \"3af45c32-632e-4dbb-8f65-9a30f443b58f\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:30:03.176278 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:03.176248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vks4c\"" Jul 07 16:30:03.184258 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:03.184234 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" Jul 07 16:30:03.317052 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:03.317025 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh"] Jul 07 16:30:03.318654 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:30:03.318627 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af45c32_632e_4dbb_8f65_9a30f443b58f.slice/crio-ab0380c6c312a0c7206aff7a9cb0f759f66cd8b5236b6d5b592150bc6240b1a9 WatchSource:0}: Error finding container ab0380c6c312a0c7206aff7a9cb0f759f66cd8b5236b6d5b592150bc6240b1a9: Status 404 returned error can't find the container with id ab0380c6c312a0c7206aff7a9cb0f759f66cd8b5236b6d5b592150bc6240b1a9 Jul 07 16:30:03.681301 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:03.681272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" event={"ID":"3af45c32-632e-4dbb-8f65-9a30f443b58f","Type":"ContainerStarted","Data":"ab0380c6c312a0c7206aff7a9cb0f759f66cd8b5236b6d5b592150bc6240b1a9"} Jul 07 16:30:04.687342 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:04.687305 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" event={"ID":"3af45c32-632e-4dbb-8f65-9a30f443b58f","Type":"ContainerStarted","Data":"010784755c888b62129670d1489f098a236c2d769d6df17e06bf67a4a681adef"} Jul 07 16:30:04.714029 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:04.713970 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5qcwh" podStartSLOduration=401.737722368 podStartE2EDuration="6m42.713953352s" totalImagesPullingTime="976.230984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:30:04.711980013 +0000 UTC m=+409.020819292" watchObservedRunningTime="2026-07-07 16:30:04.713953352 +0000 UTC m=+409.022792685" Jul 07 16:30:04.842063 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:04.842036 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:04.919645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:04.919623 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:05.064266 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:05.064201 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:07.622473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:07.622436 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:30:07.622989 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:07.622897 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:30:07.632965 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:07.632932 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:30:07.703743 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:07.703715 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:30:07.937815 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:07.937730 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:08.040120 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:08.040086 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:08.109821 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:08.109796 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:28.564478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:28.564436 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:30:28.638223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:30:28.638192 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:03.193880 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:03.193793 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:03.284169 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:03.284146 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:04.474225 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.474194 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:31:04.478738 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.478711 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.491443 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.491390 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:31:04.578766 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.578735 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:04.613868 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.613822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.613983 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.613876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.613983 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.613901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnr8\" (UniqueName: \"kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.614082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.614031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.614082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.614065 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.614148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.614082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.614148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.614100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.682694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.682669 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:04.714843 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.714946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.714946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.714946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.714946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.715151 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.715151 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.714989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnr8\" (UniqueName: \"kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.715656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.715631 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.715753 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.715631 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.715753 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.715683 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.716419 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.716397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.717436 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.717419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.717708 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.717688 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.724141 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.724118 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnr8\" (UniqueName: \"kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8\") pod \"console-5d5ffdd8fd-vwhht\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.781794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.781741 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:04.792711 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.792694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:04.922354 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.922319 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:31:04.925449 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:31:04.925416 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e23d1b_7ccb_49e6_976f_668876876226.slice/crio-e0a73b9ee0c2927bbdc0428541f7c650007f9a3347c8c34653e8f92a9187b2a5 WatchSource:0}: Error finding container e0a73b9ee0c2927bbdc0428541f7c650007f9a3347c8c34653e8f92a9187b2a5: Status 404 returned error can't find the container with id e0a73b9ee0c2927bbdc0428541f7c650007f9a3347c8c34653e8f92a9187b2a5 Jul 07 16:31:04.940659 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:04.940634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5ffdd8fd-vwhht" event={"ID":"80e23d1b-7ccb-49e6-976f-668876876226","Type":"ContainerStarted","Data":"e0a73b9ee0c2927bbdc0428541f7c650007f9a3347c8c34653e8f92a9187b2a5"} Jul 07 16:31:05.496113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:05.496084 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:05.617261 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:05.617231 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:05.946724 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:05.946694 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5ffdd8fd-vwhht" event={"ID":"80e23d1b-7ccb-49e6-976f-668876876226","Type":"ContainerStarted","Data":"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d"} Jul 07 16:31:05.967019 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:05.966976 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5ffdd8fd-vwhht" podStartSLOduration=1.9669637199999999 podStartE2EDuration="1.96696372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:31:05.965681686 +0000 UTC m=+470.274520966" watchObservedRunningTime="2026-07-07 16:31:05.96696372 +0000 UTC m=+470.275802997" Jul 07 16:31:14.792950 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:14.792911 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:14.793457 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:14.792976 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:14.797943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:14.797916 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:14.987187 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:14.987163 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:31:15.071488 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:15.071408 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:31:15.307141 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:15.307103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:15.490444 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:15.490410 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:19.872192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:19.872154 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:19.970559 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:19.970527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:25.828617 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:25.828583 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:40.092165 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.092120 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-67bf77fd7-pdnwr" podUID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" containerName="console" containerID="cri-o://cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4" gracePeriod=15 Jul 07 16:31:40.336606 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.336582 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:31:40.466692 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466632 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.466692 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466661 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.466692 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466683 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.466922 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.466922 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466719 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.466922 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466871 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wpkz\" (UniqueName: \"kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.467079 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.466927 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert\") pod \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\" (UID: \"ea2aa092-6044-42ad-bad2-3cc5a2055bde\") " Jul 07 16:31:40.467136 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467107 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca" (OuterVolumeSpecName: "service-ca") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:31:40.467136 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467124 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:31:40.467237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467149 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:31:40.467237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467158 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config" (OuterVolumeSpecName: "console-config") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:31:40.467237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467209 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.467237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.467227 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-trusted-ca-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.469005 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.468980 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:31:40.469121 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.469060 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz" (OuterVolumeSpecName: "kube-api-access-4wpkz") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "kube-api-access-4wpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:31:40.469170 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.469148 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ea2aa092-6044-42ad-bad2-3cc5a2055bde" (UID: "ea2aa092-6044-42ad-bad2-3cc5a2055bde"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:31:40.568536 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.568512 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-oauth-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.568536 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.568535 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-oauth-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.568653 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.568543 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.568653 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.568551 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpkz\" (UniqueName: \"kubernetes.io/projected/ea2aa092-6044-42ad-bad2-3cc5a2055bde-kube-api-access-4wpkz\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.568653 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.568559 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ea2aa092-6044-42ad-bad2-3cc5a2055bde-console-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:31:40.797174 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.797119 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:40.899554 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:40.899531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:31:41.092635 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.092610 2569 generic.go:356] "Generic (PLEG): container finished" podID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" containerID="cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4" exitCode=0 Jul 07 16:31:41.092946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.092659 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67bf77fd7-pdnwr" Jul 07 16:31:41.092946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.092661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67bf77fd7-pdnwr" event={"ID":"ea2aa092-6044-42ad-bad2-3cc5a2055bde","Type":"ContainerDied","Data":"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4"} Jul 07 16:31:41.092946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.092790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67bf77fd7-pdnwr" event={"ID":"ea2aa092-6044-42ad-bad2-3cc5a2055bde","Type":"ContainerDied","Data":"081318b9221b2c4b0ffc14a0e867c407c783e61aa6ca7d3408297e1f54e2a974"} Jul 07 16:31:41.092946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.092816 2569 scope.go:117] "RemoveContainer" containerID="cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4" Jul 07 16:31:41.101321 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.101304 2569 scope.go:117] "RemoveContainer" containerID="cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4" Jul 07 16:31:41.101590 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:31:41.101570 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4\": container with ID starting with cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4 not found: ID does not exist" containerID="cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4" Jul 07 16:31:41.101631 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.101599 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4"} err="failed to get container status \"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4\": rpc error: code = NotFound desc = could not find container \"cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4\": container with ID starting with cc52efcdaf9a90bfc17e476b43edd001ce736a89b4886d9008c72dd4b30d45f4 not found: ID does not exist" Jul 07 16:31:41.114645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.114623 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:31:41.119655 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.119635 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67bf77fd7-pdnwr"] Jul 07 16:31:41.885781 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.885749 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr"] Jul 07 16:31:41.886144 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.886127 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" containerName="console" Jul 07 16:31:41.886223 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.886146 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" containerName="console" Jul 07 16:31:41.886280 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.886252 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" containerName="console" Jul 07 16:31:41.890629 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.890609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:41.893762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.893735 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 16:31:41.893762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.893751 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 16:31:41.894569 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.894552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5mvw2\"" Jul 07 16:31:41.908381 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.908330 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr"] Jul 07 16:31:41.978448 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.978420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:41.978541 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.978466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:41.978541 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:41.978498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2l9\" (UniqueName: \"kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.079043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.079007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.079043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.079051 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2l9\" (UniqueName: \"kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.079180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.079097 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.079362 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.079343 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.079406 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.079366 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.088721 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.088699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2l9\" (UniqueName: \"kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.200098 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.200041 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:31:42.272019 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.269639 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2aa092-6044-42ad-bad2-3cc5a2055bde" path="/var/lib/kubelet/pods/ea2aa092-6044-42ad-bad2-3cc5a2055bde/volumes" Jul 07 16:31:42.331138 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:42.331079 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr"] Jul 07 16:31:42.333687 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:31:42.333662 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69644ac9_e072_4c25_a6b3_108d58a48c66.slice/crio-4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6 WatchSource:0}: Error finding container 4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6: Status 404 returned error can't find the container with id 4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6 Jul 07 16:31:43.103549 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:43.103514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" event={"ID":"69644ac9-e072-4c25-a6b3-108d58a48c66","Type":"ContainerStarted","Data":"4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6"} Jul 07 16:31:54.158742 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:54.158701 2569 generic.go:356] "Generic (PLEG): container finished" podID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerID="307abce7738a6cebace654ea88069c1f8c5790aacef21d1e4412a29e2be3186f" exitCode=0 Jul 07 16:31:54.159214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:54.158765 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" event={"ID":"69644ac9-e072-4c25-a6b3-108d58a48c66","Type":"ContainerDied","Data":"307abce7738a6cebace654ea88069c1f8c5790aacef21d1e4412a29e2be3186f"} Jul 07 16:31:56.171294 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:56.171269 2569 generic.go:356] "Generic (PLEG): container finished" podID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerID="d2e56140f167af5d7c280e6f3b6acb52ff76ac81a31d86b7bd30fa9a9a139b2c" exitCode=0 Jul 07 16:31:56.171678 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:31:56.171320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" event={"ID":"69644ac9-e072-4c25-a6b3-108d58a48c66","Type":"ContainerDied","Data":"d2e56140f167af5d7c280e6f3b6acb52ff76ac81a31d86b7bd30fa9a9a139b2c"} Jul 07 16:32:06.216162 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:06.216128 2569 generic.go:356] "Generic (PLEG): container finished" podID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerID="4a85708fc06cc645e9c2ba6e99e8a537320e0683556f2fcefa73ca3c72cfbec8" exitCode=0 Jul 07 16:32:06.216476 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:06.216189 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" event={"ID":"69644ac9-e072-4c25-a6b3-108d58a48c66","Type":"ContainerDied","Data":"4a85708fc06cc645e9c2ba6e99e8a537320e0683556f2fcefa73ca3c72cfbec8"} Jul 07 16:32:07.346844 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.346809 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:32:07.370957 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.370929 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h2l9\" (UniqueName: \"kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9\") pod \"69644ac9-e072-4c25-a6b3-108d58a48c66\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " Jul 07 16:32:07.371088 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.371022 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util\") pod \"69644ac9-e072-4c25-a6b3-108d58a48c66\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " Jul 07 16:32:07.371088 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.371057 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle\") pod \"69644ac9-e072-4c25-a6b3-108d58a48c66\" (UID: \"69644ac9-e072-4c25-a6b3-108d58a48c66\") " Jul 07 16:32:07.371757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.371732 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle" (OuterVolumeSpecName: "bundle") pod "69644ac9-e072-4c25-a6b3-108d58a48c66" (UID: "69644ac9-e072-4c25-a6b3-108d58a48c66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:07.373078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.373055 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9" (OuterVolumeSpecName: "kube-api-access-2h2l9") pod "69644ac9-e072-4c25-a6b3-108d58a48c66" (UID: "69644ac9-e072-4c25-a6b3-108d58a48c66"). InnerVolumeSpecName "kube-api-access-2h2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:32:07.374995 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.374971 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util" (OuterVolumeSpecName: "util") pod "69644ac9-e072-4c25-a6b3-108d58a48c66" (UID: "69644ac9-e072-4c25-a6b3-108d58a48c66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:07.472494 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.472432 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2l9\" (UniqueName: \"kubernetes.io/projected/69644ac9-e072-4c25-a6b3-108d58a48c66-kube-api-access-2h2l9\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:07.472494 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.472466 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:07.472494 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:07.472476 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69644ac9-e072-4c25-a6b3-108d58a48c66-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:08.227301 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:08.227232 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" event={"ID":"69644ac9-e072-4c25-a6b3-108d58a48c66","Type":"ContainerDied","Data":"4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6"} Jul 07 16:32:08.227301 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:08.227278 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qlnbr" Jul 07 16:32:08.227481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:08.227318 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fcc459b94d9867ae747b50c2593bf4135d70d3176c191e8c7209c0d8ed284e6" Jul 07 16:32:13.496545 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:13.496515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:14.756297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:14.756268 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:16.938328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:16.938296 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:17.043950 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:17.043921 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:19.110696 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:19.110662 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:19.221237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:19.221212 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:19.520699 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:19.520617 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:19.702850 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:19.702809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:21.529010 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.528973 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.697436 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8cxkd"] Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698277 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="pull" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698301 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="pull" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698351 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="util" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698360 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="util" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698393 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="extract" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698402 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="extract" Jul 07 16:32:21.703816 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.698722 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69644ac9-e072-4c25-a6b3-108d58a48c66" containerName="extract" Jul 07 16:32:21.705321 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.705300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.707948 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.707923 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 16:32:21.708069 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.708051 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 16:32:21.708993 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.708969 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g52kx\"" Jul 07 16:32:21.713241 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.713212 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8cxkd"] Jul 07 16:32:21.737818 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.737798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:21.775049 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.775027 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.775122 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.775086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmxf\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-kube-api-access-smmxf\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.876256 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.876233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmxf\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-kube-api-access-smmxf\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.876325 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.876272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.885396 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.885370 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:21.885479 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:21.885447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmxf\" (UniqueName: \"kubernetes.io/projected/79b82773-7852-47b7-ad88-0b94dcfdfa7d-kube-api-access-smmxf\") pod \"cert-manager-webhook-6486ffd997-8cxkd\" (UID: \"79b82773-7852-47b7-ad88-0b94dcfdfa7d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:22.031099 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.031076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:22.165821 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.165792 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8cxkd"] Jul 07 16:32:22.167633 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:32:22.167607 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79b82773_7852_47b7_ad88_0b94dcfdfa7d.slice/crio-38e57fbaf1db3e705a1c1bd456838dff129a2c5d5e96f31c70a3993135dc22e2 WatchSource:0}: Error finding container 38e57fbaf1db3e705a1c1bd456838dff129a2c5d5e96f31c70a3993135dc22e2: Status 404 returned error can't find the container with id 38e57fbaf1db3e705a1c1bd456838dff129a2c5d5e96f31c70a3993135dc22e2 Jul 07 16:32:22.282676 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.282651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" event={"ID":"79b82773-7852-47b7-ad88-0b94dcfdfa7d","Type":"ContainerStarted","Data":"38e57fbaf1db3e705a1c1bd456838dff129a2c5d5e96f31c70a3993135dc22e2"} Jul 07 16:32:22.968504 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.968474 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rstzg"] Jul 07 16:32:22.971619 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.971601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:22.974216 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.974194 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fmq4k\"" Jul 07 16:32:22.982153 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:22.982132 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rstzg"] Jul 07 16:32:23.087148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.087121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brl6n\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-kube-api-access-brl6n\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.087271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.087198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.188558 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.188527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.188649 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.188569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brl6n\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-kube-api-access-brl6n\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.196658 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.196635 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.196770 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.196750 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brl6n\" (UniqueName: \"kubernetes.io/projected/cffcc215-f16f-4955-8a0f-a3697227e784-kube-api-access-brl6n\") pod \"cert-manager-cainjector-559cdd8c87-rstzg\" (UID: \"cffcc215-f16f-4955-8a0f-a3697227e784\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.282314 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.282262 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" Jul 07 16:32:23.416727 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:23.416584 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rstzg"] Jul 07 16:32:23.418792 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:32:23.418763 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcffcc215_f16f_4955_8a0f_a3697227e784.slice/crio-b41fe7853be37f1f881268bf4c1195ef3833e2e42d6a9e602c96c7cb8e2fab71 WatchSource:0}: Error finding container b41fe7853be37f1f881268bf4c1195ef3833e2e42d6a9e602c96c7cb8e2fab71: Status 404 returned error can't find the container with id b41fe7853be37f1f881268bf4c1195ef3833e2e42d6a9e602c96c7cb8e2fab71 Jul 07 16:32:24.296771 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:24.296732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" event={"ID":"cffcc215-f16f-4955-8a0f-a3697227e784","Type":"ContainerStarted","Data":"b41fe7853be37f1f881268bf4c1195ef3833e2e42d6a9e602c96c7cb8e2fab71"} Jul 07 16:32:24.783527 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:24.783492 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:24.853583 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:24.853523 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:26.307648 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:26.307613 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" event={"ID":"79b82773-7852-47b7-ad88-0b94dcfdfa7d","Type":"ContainerStarted","Data":"d480efc2b6c8e729b932c2673590b72a9dbe4ae3b9f1651bf7636f0d73711d33"} Jul 07 16:32:26.308130 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:26.307660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:26.309188 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:26.309156 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" event={"ID":"cffcc215-f16f-4955-8a0f-a3697227e784","Type":"ContainerStarted","Data":"1626b317a6d9b493001cb77333cdc5de420dfc17f05e038a1be1c6893a3b5cd2"} Jul 07 16:32:26.328055 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:26.327994 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" podStartSLOduration=2.094318139 podStartE2EDuration="5.327976084s" totalImagesPullingTime="3.233657945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:32:26.326824654 +0000 UTC m=+550.635663933" watchObservedRunningTime="2026-07-07 16:32:26.327976084 +0000 UTC m=+550.636815363" Jul 07 16:32:26.344658 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:26.344620 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rstzg" podStartSLOduration=2.356308693 podStartE2EDuration="4.344606501s" totalImagesPullingTime="1.988297808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:32:26.343578442 +0000 UTC m=+550.652417722" watchObservedRunningTime="2026-07-07 16:32:26.344606501 +0000 UTC m=+550.653445780" Jul 07 16:32:28.727752 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:28.727722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:32.315072 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:32.315043 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8cxkd" Jul 07 16:32:33.834679 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:33.834639 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:34.565131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.565096 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz"] Jul 07 16:32:34.568332 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.568308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.570914 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.570890 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 16:32:34.570996 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.570900 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 16:32:34.571885 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.571861 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5mvw2\"" Jul 07 16:32:34.578374 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.578347 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz"] Jul 07 16:32:34.673686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.673663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.673786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.673701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.673786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.673722 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6vg\" (UniqueName: \"kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.774585 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.774562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.774656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.774590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6vg\" (UniqueName: \"kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.774656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.774642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.775020 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.775005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.775068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.775005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.784549 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.784526 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6vg\" (UniqueName: \"kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:34.879517 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:34.879498 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:35.005736 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:35.005707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz"] Jul 07 16:32:35.007947 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:32:35.007918 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fd147f_48b3_446b_8fe4_2d972de0ed30.slice/crio-da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27 WatchSource:0}: Error finding container da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27: Status 404 returned error can't find the container with id da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27 Jul 07 16:32:35.351582 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:35.351546 2569 generic.go:356] "Generic (PLEG): container finished" podID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerID="dbf843016f88faf7150eb654083d3a9ba6de7cbbc473582c448ef46030bf6129" exitCode=0 Jul 07 16:32:35.351705 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:35.351634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" event={"ID":"36fd147f-48b3-446b-8fe4-2d972de0ed30","Type":"ContainerDied","Data":"dbf843016f88faf7150eb654083d3a9ba6de7cbbc473582c448ef46030bf6129"} Jul 07 16:32:35.351705 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:35.351692 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" event={"ID":"36fd147f-48b3-446b-8fe4-2d972de0ed30","Type":"ContainerStarted","Data":"da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27"} Jul 07 16:32:38.369190 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:38.369154 2569 generic.go:356] "Generic (PLEG): container finished" podID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerID="2cdf753d52c34dc3ca44ee521af3bc46f366837a1bf14557e1fc2b45e35f6f48" exitCode=0 Jul 07 16:32:38.369592 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:38.369213 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" event={"ID":"36fd147f-48b3-446b-8fe4-2d972de0ed30","Type":"ContainerDied","Data":"2cdf753d52c34dc3ca44ee521af3bc46f366837a1bf14557e1fc2b45e35f6f48"} Jul 07 16:32:39.376515 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:39.376479 2569 generic.go:356] "Generic (PLEG): container finished" podID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerID="8a2b15b37b898755b01aa46ceefe97575fd3e3f5443a3be852901d112629dc82" exitCode=0 Jul 07 16:32:39.376927 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:39.376541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" event={"ID":"36fd147f-48b3-446b-8fe4-2d972de0ed30","Type":"ContainerDied","Data":"8a2b15b37b898755b01aa46ceefe97575fd3e3f5443a3be852901d112629dc82"} Jul 07 16:32:40.501367 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.501340 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:40.619826 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.619796 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle\") pod \"36fd147f-48b3-446b-8fe4-2d972de0ed30\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " Jul 07 16:32:40.619975 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.619849 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn6vg\" (UniqueName: \"kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg\") pod \"36fd147f-48b3-446b-8fe4-2d972de0ed30\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " Jul 07 16:32:40.619975 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.619883 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util\") pod \"36fd147f-48b3-446b-8fe4-2d972de0ed30\" (UID: \"36fd147f-48b3-446b-8fe4-2d972de0ed30\") " Jul 07 16:32:40.620250 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.620226 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle" (OuterVolumeSpecName: "bundle") pod "36fd147f-48b3-446b-8fe4-2d972de0ed30" (UID: "36fd147f-48b3-446b-8fe4-2d972de0ed30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:40.621890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.621867 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg" (OuterVolumeSpecName: "kube-api-access-rn6vg") pod "36fd147f-48b3-446b-8fe4-2d972de0ed30" (UID: "36fd147f-48b3-446b-8fe4-2d972de0ed30"). InnerVolumeSpecName "kube-api-access-rn6vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:32:40.624592 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.624569 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util" (OuterVolumeSpecName: "util") pod "36fd147f-48b3-446b-8fe4-2d972de0ed30" (UID: "36fd147f-48b3-446b-8fe4-2d972de0ed30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:40.720810 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.720741 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:40.720810 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.720774 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6vg\" (UniqueName: \"kubernetes.io/projected/36fd147f-48b3-446b-8fe4-2d972de0ed30-kube-api-access-rn6vg\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:40.720810 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:40.720787 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36fd147f-48b3-446b-8fe4-2d972de0ed30-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:41.387221 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:41.387194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" event={"ID":"36fd147f-48b3-446b-8fe4-2d972de0ed30","Type":"ContainerDied","Data":"da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27"} Jul 07 16:32:41.387400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:41.387239 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6f48bf76c6cc6615cd39c9cdc43c983bdae070528e13be829cd6bcac210d27" Jul 07 16:32:41.387400 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:41.387210 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbz5tz" Jul 07 16:32:45.501807 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:45.501774 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:45.599083 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:45.599054 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:48.455680 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:48.455650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:48.521074 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:48.521047 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:51.750383 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:51.750347 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:53.223360 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.223317 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:53.402273 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402233 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc"] Jul 07 16:32:53.402588 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402573 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="util" Jul 07 16:32:53.402638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402590 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="util" Jul 07 16:32:53.402638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402600 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="pull" Jul 07 16:32:53.402638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402606 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="pull" Jul 07 16:32:53.402638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402627 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="extract" Jul 07 16:32:53.402638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402632 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="extract" Jul 07 16:32:53.402808 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.402699 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36fd147f-48b3-446b-8fe4-2d972de0ed30" containerName="extract" Jul 07 16:32:53.407264 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.407241 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.409633 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.409610 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 16:32:53.409730 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.409610 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 16:32:53.410591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.410570 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5mvw2\"" Jul 07 16:32:53.416155 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.416130 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc"] Jul 07 16:32:53.510257 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.510199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77tgh\" (UniqueName: \"kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.510257 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.510232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.510380 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.510297 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.611344 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.611316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77tgh\" (UniqueName: \"kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.611445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.611355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.611445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.611399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.611755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.611738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.611798 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.611781 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.620429 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.620404 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77tgh\" (UniqueName: \"kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.684620 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.684600 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:53.718436 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.718413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:53.846815 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:53.846787 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc"] Jul 07 16:32:53.850038 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:32:53.850004 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f0563e_acfa_4cef_8b3c_99b7e954694e.slice/crio-f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f WatchSource:0}: Error finding container f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f: Status 404 returned error can't find the container with id f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f Jul 07 16:32:54.450974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:54.450941 2569 generic.go:356] "Generic (PLEG): container finished" podID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerID="82b1c5e35c32c8fffd024039cb5497fb8fb72e5828c82023d97b81f83a8e4e34" exitCode=0 Jul 07 16:32:54.451375 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:54.451031 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" event={"ID":"55f0563e-acfa-4cef-8b3c-99b7e954694e","Type":"ContainerDied","Data":"82b1c5e35c32c8fffd024039cb5497fb8fb72e5828c82023d97b81f83a8e4e34"} Jul 07 16:32:54.451375 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:54.451094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" event={"ID":"55f0563e-acfa-4cef-8b3c-99b7e954694e","Type":"ContainerStarted","Data":"f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f"} Jul 07 16:32:55.457512 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:55.457484 2569 generic.go:356] "Generic (PLEG): container finished" podID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerID="90c17939c439a193506b58790af88427a9c7f912b5716e5e99712c0adbb7603f" exitCode=0 Jul 07 16:32:55.457910 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:55.457562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" event={"ID":"55f0563e-acfa-4cef-8b3c-99b7e954694e","Type":"ContainerDied","Data":"90c17939c439a193506b58790af88427a9c7f912b5716e5e99712c0adbb7603f"} Jul 07 16:32:56.464749 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:56.464714 2569 generic.go:356] "Generic (PLEG): container finished" podID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerID="ee687426b08a4df4a8704b541b57cf9eb9eb9c5f4ce4c8a3d8fb7d71221c61a0" exitCode=0 Jul 07 16:32:56.465181 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:56.464810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" event={"ID":"55f0563e-acfa-4cef-8b3c-99b7e954694e","Type":"ContainerDied","Data":"ee687426b08a4df4a8704b541b57cf9eb9eb9c5f4ce4c8a3d8fb7d71221c61a0"} Jul 07 16:32:57.605415 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.605387 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:57.743209 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.743149 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77tgh\" (UniqueName: \"kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh\") pod \"55f0563e-acfa-4cef-8b3c-99b7e954694e\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " Jul 07 16:32:57.743324 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.743213 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle\") pod \"55f0563e-acfa-4cef-8b3c-99b7e954694e\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " Jul 07 16:32:57.743324 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.743252 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util\") pod \"55f0563e-acfa-4cef-8b3c-99b7e954694e\" (UID: \"55f0563e-acfa-4cef-8b3c-99b7e954694e\") " Jul 07 16:32:57.743915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.743892 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle" (OuterVolumeSpecName: "bundle") pod "55f0563e-acfa-4cef-8b3c-99b7e954694e" (UID: "55f0563e-acfa-4cef-8b3c-99b7e954694e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:57.745128 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.745107 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh" (OuterVolumeSpecName: "kube-api-access-77tgh") pod "55f0563e-acfa-4cef-8b3c-99b7e954694e" (UID: "55f0563e-acfa-4cef-8b3c-99b7e954694e"). InnerVolumeSpecName "kube-api-access-77tgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:32:57.749316 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.749294 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util" (OuterVolumeSpecName: "util") pod "55f0563e-acfa-4cef-8b3c-99b7e954694e" (UID: "55f0563e-acfa-4cef-8b3c-99b7e954694e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:32:57.844476 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.844452 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:57.844476 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.844472 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55f0563e-acfa-4cef-8b3c-99b7e954694e-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:57.844601 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:57.844481 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77tgh\" (UniqueName: \"kubernetes.io/projected/55f0563e-acfa-4cef-8b3c-99b7e954694e-kube-api-access-77tgh\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:32:58.035561 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.035483 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:58.113855 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.113812 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq"] Jul 07 16:32:58.114177 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114164 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="extract" Jul 07 16:32:58.114224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114182 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="extract" Jul 07 16:32:58.114224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114203 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="util" Jul 07 16:32:58.114224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114208 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="util" Jul 07 16:32:58.114224 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114220 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="pull" Jul 07 16:32:58.114336 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114225 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="pull" Jul 07 16:32:58.114336 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.114285 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f0563e-acfa-4cef-8b3c-99b7e954694e" containerName="extract" Jul 07 16:32:58.116300 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.116285 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.120582 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.120561 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 16:32:58.121539 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.121521 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 16:32:58.121657 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.121564 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9z65z\"" Jul 07 16:32:58.121768 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.121753 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 16:32:58.121861 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.121760 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 16:32:58.121861 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.121808 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 16:32:58.136300 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.136278 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq"] Jul 07 16:32:58.163615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.163594 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:32:58.249063 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.249042 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-manager-config\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.249185 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.249087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.249185 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.249110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-metrics-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.249329 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.249209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wgs\" (UniqueName: \"kubernetes.io/projected/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-kube-api-access-w9wgs\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.350007 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.349984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-manager-config\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.350131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.350031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.350131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.350047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-metrics-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.350131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.350088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wgs\" (UniqueName: \"kubernetes.io/projected/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-kube-api-access-w9wgs\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.350671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.350648 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-manager-config\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.352510 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.352484 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.352684 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.352662 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-metrics-cert\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.361520 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.361501 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wgs\" (UniqueName: \"kubernetes.io/projected/f4719801-3bbe-4bf3-bb73-1ff76c2d93ae-kube-api-access-w9wgs\") pod \"lws-controller-manager-778798b69f-vxtmq\" (UID: \"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae\") " pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.426480 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.426457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:32:58.477803 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.477767 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" event={"ID":"55f0563e-acfa-4cef-8b3c-99b7e954694e","Type":"ContainerDied","Data":"f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f"} Jul 07 16:32:58.477929 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.477790 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8ddkc" Jul 07 16:32:58.477929 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.477876 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f584a9f6a484352bdc52c6ca4a9b5293a15cf13a9f236a2637d695ee9ce9731f" Jul 07 16:32:58.579213 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:58.579190 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq"] Jul 07 16:32:58.581236 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:32:58.581210 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4719801_3bbe_4bf3_bb73_1ff76c2d93ae.slice/crio-995f5e37dd38240a5f75b0ffa2062a19d81903f4a0f67e551b356498cd8f38e4 WatchSource:0}: Error finding container 995f5e37dd38240a5f75b0ffa2062a19d81903f4a0f67e551b356498cd8f38e4: Status 404 returned error can't find the container with id 995f5e37dd38240a5f75b0ffa2062a19d81903f4a0f67e551b356498cd8f38e4 Jul 07 16:32:59.485254 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:32:59.485218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" event={"ID":"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae","Type":"ContainerStarted","Data":"995f5e37dd38240a5f75b0ffa2062a19d81903f4a0f67e551b356498cd8f38e4"} Jul 07 16:33:01.181592 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:01.181558 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:02.500458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:02.500421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" event={"ID":"f4719801-3bbe-4bf3-bb73-1ff76c2d93ae","Type":"ContainerStarted","Data":"2780cb862babcdec4a568eed56cfc973f83db7f594c396cffe65ff821e7e84ac"} Jul 07 16:33:02.500875 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:02.500497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:33:02.524128 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:02.524080 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" podStartSLOduration=1.038805829 podStartE2EDuration="4.524064518s" totalImagesPullingTime="3.485258689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:02.523467349 +0000 UTC m=+586.832306627" watchObservedRunningTime="2026-07-07 16:33:02.524064518 +0000 UTC m=+586.832903795" Jul 07 16:33:03.758068 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:03.758036 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:03.946035 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:03.945989 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:09.426439 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:09.426411 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:09.522652 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:09.522627 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:09.993026 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:09.993000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.264894 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.264795 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.380083 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.380041 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.426083 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.426051 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9"] Jul 07 16:33:10.428531 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.428511 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.431477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.431454 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 16:33:10.431577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.431528 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 16:33:10.431912 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.431892 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 16:33:10.432012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.431896 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 16:33:10.432012 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.431997 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9fsmk\"" Jul 07 16:33:10.449658 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.449634 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9"] Jul 07 16:33:10.540797 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.540732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.540797 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.540769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.540961 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.540816 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hq8f\" (UniqueName: \"kubernetes.io/projected/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-kube-api-access-4hq8f\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.540961 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.540857 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.540961 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.540917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.632121 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.632097 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.642190 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hq8f\" (UniqueName: \"kubernetes.io/projected/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-kube-api-access-4hq8f\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.642297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.642297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.642297 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.642461 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.642872 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.642824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-manifests\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.644663 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.644639 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-tmp\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.644763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.644709 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.644763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.644736 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.651481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.651462 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hq8f\" (UniqueName: \"kubernetes.io/projected/ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766-kube-api-access-4hq8f\") pod \"opendatahub-operator-controller-manager-7fb6d99899-cwsf9\" (UID: \"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766\") " pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.733053 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.733030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.739102 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.739080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:10.832535 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.832509 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:10.884081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.884057 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9"] Jul 07 16:33:10.885450 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:33:10.885424 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4474f3_0f1e_4e32_a2c7_3a12cc3bd766.slice/crio-03ef97a9a7e951a90ef49f8c6b3ba60c469726a4be7223df520957114ede416f WatchSource:0}: Error finding container 03ef97a9a7e951a90ef49f8c6b3ba60c469726a4be7223df520957114ede416f: Status 404 returned error can't find the container with id 03ef97a9a7e951a90ef49f8c6b3ba60c469726a4be7223df520957114ede416f Jul 07 16:33:10.932085 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:10.932058 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:11.031294 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:11.031267 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:11.131996 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:11.131971 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:11.544156 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:11.544074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" event={"ID":"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766","Type":"ContainerStarted","Data":"03ef97a9a7e951a90ef49f8c6b3ba60c469726a4be7223df520957114ede416f"} Jul 07 16:33:13.507745 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:13.507715 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778798b69f-vxtmq" Jul 07 16:33:14.564878 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:14.564827 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" event={"ID":"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766","Type":"ContainerStarted","Data":"db26f36228d24058703c4fb210920ad29c56d1e606b8b19a9405fa3876cc7613"} Jul 07 16:33:15.570989 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:15.570910 2569 generic.go:356] "Generic (PLEG): container finished" podID="ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766" containerID="db26f36228d24058703c4fb210920ad29c56d1e606b8b19a9405fa3876cc7613" exitCode=0 Jul 07 16:33:15.570989 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:15.570955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" event={"ID":"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766","Type":"ContainerDied","Data":"db26f36228d24058703c4fb210920ad29c56d1e606b8b19a9405fa3876cc7613"} Jul 07 16:33:16.125020 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:16.124995 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:33:16.125828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:16.125802 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:33:16.577458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:16.577427 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" event={"ID":"ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766","Type":"ContainerStarted","Data":"6c81f673d055a63f61a54896d10d5d209e71c6d7044a43caa685d5000e32397a"} Jul 07 16:33:16.577813 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:16.577556 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:16.602882 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:16.602819 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" podStartSLOduration=1.951392816 podStartE2EDuration="6.602806095s" totalImagesPullingTime="3.608316321s" totalInitContainerRuntime="1.043096958s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:16.600204667 +0000 UTC m=+600.909043943" watchObservedRunningTime="2026-07-07 16:33:16.602806095 +0000 UTC m=+600.911645373" Jul 07 16:33:21.914684 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:21.914649 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:27.584617 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:27.584572 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fb6d99899-cwsf9" Jul 07 16:33:29.916176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:29.916141 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:30.067682 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.067644 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:30.677542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.677515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:30.899279 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.899249 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5"] Jul 07 16:33:30.973761 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.973695 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5"] Jul 07 16:33:30.974132 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.973780 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:30.980262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.980083 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zz628\"" Jul 07 16:33:30.980262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.980233 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 16:33:30.981479 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.981459 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 16:33:30.981590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.981540 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 16:33:30.981590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.981574 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 16:33:30.998098 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.998073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9827d53c-ce10-4438-b9d5-ccd350c49338-tmp\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:30.998192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.998118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4w9j\" (UniqueName: \"kubernetes.io/projected/9827d53c-ce10-4438-b9d5-ccd350c49338-kube-api-access-d4w9j\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:30.998192 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:30.998161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9827d53c-ce10-4438-b9d5-ccd350c49338-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.045359 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.045334 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:31.099512 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.099489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9827d53c-ce10-4438-b9d5-ccd350c49338-tmp\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.099638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.099554 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4w9j\" (UniqueName: \"kubernetes.io/projected/9827d53c-ce10-4438-b9d5-ccd350c49338-kube-api-access-d4w9j\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.099638 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.099589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9827d53c-ce10-4438-b9d5-ccd350c49338-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.101644 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.101616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9827d53c-ce10-4438-b9d5-ccd350c49338-tmp\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.102343 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.102323 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9827d53c-ce10-4438-b9d5-ccd350c49338-tls-certs\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.121056 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.121030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4w9j\" (UniqueName: \"kubernetes.io/projected/9827d53c-ce10-4438-b9d5-ccd350c49338-kube-api-access-d4w9j\") pod \"kube-auth-proxy-6dfd697d7-mmjx5\" (UID: \"9827d53c-ce10-4438-b9d5-ccd350c49338\") " pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.166676 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.166641 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:31.283154 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.283098 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" Jul 07 16:33:31.327544 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.327520 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:31.459328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.459297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:31.468173 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.468137 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5"] Jul 07 16:33:31.472719 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:33:31.472691 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9827d53c_ce10_4438_b9d5_ccd350c49338.slice/crio-16beb3d75a80fe1a630367d1a5edcf27fea6865d227a37c449a700cce076357d WatchSource:0}: Error finding container 16beb3d75a80fe1a630367d1a5edcf27fea6865d227a37c449a700cce076357d: Status 404 returned error can't find the container with id 16beb3d75a80fe1a630367d1a5edcf27fea6865d227a37c449a700cce076357d Jul 07 16:33:31.474651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.474635 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:33:31.650966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:31.650941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" event={"ID":"9827d53c-ce10-4438-b9d5-ccd350c49338","Type":"ContainerStarted","Data":"16beb3d75a80fe1a630367d1a5edcf27fea6865d227a37c449a700cce076357d"} Jul 07 16:33:32.298734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:32.298700 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:32.578191 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:32.578110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:32.732222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:32.732189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:34.046355 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:34.046275 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:34.665573 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:34.665543 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" event={"ID":"9827d53c-ce10-4438-b9d5-ccd350c49338","Type":"ContainerStarted","Data":"0f1f86417018036e8f21b7b69dad94a45402263a3a5d6a59094b1da7ac26f4c2"} Jul 07 16:33:34.683986 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:34.683940 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dfd697d7-mmjx5" podStartSLOduration=1.634380407 podStartE2EDuration="4.683925014s" totalImagesPullingTime="3.049544607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:33:34.681817147 +0000 UTC m=+618.990656426" watchObservedRunningTime="2026-07-07 16:33:34.683925014 +0000 UTC m=+618.992764301" Jul 07 16:33:37.820515 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:37.820465 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:37.961855 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:37.961762 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:38.906103 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:38.906052 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:39.113272 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:39.113242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:39.296720 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:39.296651 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:41.786394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:41.786360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:42.038103 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:42.037998 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:42.332670 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:42.332645 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:42.453806 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:42.453776 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:42.544711 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:42.544677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:33:42.645369 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:33:42.645284 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:31.701109 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:31.701076 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:31.789424 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:31.789395 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:32.380227 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:32.380202 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:33.035791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:33.035759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:33.240253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:33.240220 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:43.851784 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:43.851748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:34:47.847391 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.847352 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm"] Jul 07 16:34:47.854569 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.854525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.859275 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.859248 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 16:34:47.859648 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.859619 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7cb8t\"" Jul 07 16:34:47.860546 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.860516 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 16:34:47.862880 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.862856 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm"] Jul 07 16:34:47.872609 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.872580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.872726 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.872631 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.872798 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.872720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxknx\" (UniqueName: \"kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.974098 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.974066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxknx\" (UniqueName: \"kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.974219 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.974137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.974219 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.974196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.974491 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.974470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.974576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.974534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:47.983873 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:47.983845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxknx\" (UniqueName: \"kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:48.168126 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.168074 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:48.250521 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.250490 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst"] Jul 07 16:34:48.254727 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.254504 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.273134 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.273106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst"] Jul 07 16:34:48.277342 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.277316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.277447 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.277389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lm4m\" (UniqueName: \"kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.277447 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.277424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.308591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.308567 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm"] Jul 07 16:34:48.310068 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:34:48.310042 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dcac0f_c709_4603_884d_617f73832558.slice/crio-c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3 WatchSource:0}: Error finding container c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3: Status 404 returned error can't find the container with id c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3 Jul 07 16:34:48.378062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.378030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.378183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.378080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lm4m\" (UniqueName: \"kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.378183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.378102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.378380 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.378362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.378454 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.378423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.386725 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.386702 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lm4m\" (UniqueName: \"kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.569556 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.569497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:48.647796 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.647770 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp"] Jul 07 16:34:48.651946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.651923 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.661395 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.661308 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp"] Jul 07 16:34:48.680112 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.680086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.680228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.680123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh62\" (UniqueName: \"kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.680228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.680149 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.703856 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.703734 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst"] Jul 07 16:34:48.705893 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:34:48.705871 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf473e35c_7340_4b50_8293_b2004f9f7e89.slice/crio-0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae WatchSource:0}: Error finding container 0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae: Status 404 returned error can't find the container with id 0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae Jul 07 16:34:48.781401 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.781373 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.781543 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.781407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh62\" (UniqueName: \"kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.781543 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.781431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.781720 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.781701 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.781769 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.781730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.790362 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.790323 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh62\" (UniqueName: \"kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:48.964597 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:48.964574 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:49.040650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.040618 2569 generic.go:356] "Generic (PLEG): container finished" podID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerID="0c58a0f3c75f62bde73b5546548935df66d89b71c227735e2ea0d5caf1fc3545" exitCode=0 Jul 07 16:34:49.040777 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.040696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" event={"ID":"f473e35c-7340-4b50-8293-b2004f9f7e89","Type":"ContainerDied","Data":"0c58a0f3c75f62bde73b5546548935df66d89b71c227735e2ea0d5caf1fc3545"} Jul 07 16:34:49.040777 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.040753 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" event={"ID":"f473e35c-7340-4b50-8293-b2004f9f7e89","Type":"ContainerStarted","Data":"0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae"} Jul 07 16:34:49.042998 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.042974 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2dcac0f-c709-4603-884d-617f73832558" containerID="0afa1a64ea03490d5f1bcb38a89ebee7bb688462aec5f5b1e23dd4513cdccc35" exitCode=0 Jul 07 16:34:49.043094 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.043020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" event={"ID":"f2dcac0f-c709-4603-884d-617f73832558","Type":"ContainerDied","Data":"0afa1a64ea03490d5f1bcb38a89ebee7bb688462aec5f5b1e23dd4513cdccc35"} Jul 07 16:34:49.043144 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.043100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" event={"ID":"f2dcac0f-c709-4603-884d-617f73832558","Type":"ContainerStarted","Data":"c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3"} Jul 07 16:34:49.052502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.052470 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m"] Jul 07 16:34:49.057636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.057612 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.065678 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.065599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m"] Jul 07 16:34:49.084018 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.083989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.084131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.084074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.084131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.084114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6sb\" (UniqueName: \"kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.104149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.104106 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp"] Jul 07 16:34:49.106360 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:34:49.106338 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe64e709_b1cb_4287_8bff_e6829a7e3134.slice/crio-690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c WatchSource:0}: Error finding container 690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c: Status 404 returned error can't find the container with id 690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c Jul 07 16:34:49.185143 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.185107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.185260 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.185152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.185260 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.185198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6sb\" (UniqueName: \"kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.185430 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.185409 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.185497 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.185448 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.193735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.193706 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6sb\" (UniqueName: \"kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.368665 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.368637 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:49.504763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:49.504741 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m"] Jul 07 16:34:49.505920 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:34:49.505896 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79b6b94_003f_4802_8ab4_8e4fe0c38eae.slice/crio-65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1 WatchSource:0}: Error finding container 65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1: Status 404 returned error can't find the container with id 65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1 Jul 07 16:34:50.050690 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.050662 2569 generic.go:356] "Generic (PLEG): container finished" podID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerID="2bd5658430426e189ed8c767d920f43fecc1417e45fd35269e76f36d7bce4d7a" exitCode=0 Jul 07 16:34:50.051038 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.050743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" event={"ID":"f473e35c-7340-4b50-8293-b2004f9f7e89","Type":"ContainerDied","Data":"2bd5658430426e189ed8c767d920f43fecc1417e45fd35269e76f36d7bce4d7a"} Jul 07 16:34:50.052316 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.052297 2569 generic.go:356] "Generic (PLEG): container finished" podID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerID="085d752d6ebd465578268537c9b7ed47742066dfaaf6ee4ddd5fc76e594cc5f3" exitCode=0 Jul 07 16:34:50.052406 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.052379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" event={"ID":"f79b6b94-003f-4802-8ab4-8e4fe0c38eae","Type":"ContainerDied","Data":"085d752d6ebd465578268537c9b7ed47742066dfaaf6ee4ddd5fc76e594cc5f3"} Jul 07 16:34:50.052469 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.052430 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" event={"ID":"f79b6b94-003f-4802-8ab4-8e4fe0c38eae","Type":"ContainerStarted","Data":"65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1"} Jul 07 16:34:50.054357 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.054335 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2dcac0f-c709-4603-884d-617f73832558" containerID="1bc6420fc2149c502e406a97a4425b5b944223143dc4821f624be67505263cd0" exitCode=0 Jul 07 16:34:50.054493 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.054478 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" event={"ID":"f2dcac0f-c709-4603-884d-617f73832558","Type":"ContainerDied","Data":"1bc6420fc2149c502e406a97a4425b5b944223143dc4821f624be67505263cd0"} Jul 07 16:34:50.056185 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.056160 2569 generic.go:356] "Generic (PLEG): container finished" podID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerID="bdd5ca02afc1bb9de940833932bba7b1186cc6369df5a8563adbd4048c39479e" exitCode=0 Jul 07 16:34:50.056271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.056195 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" event={"ID":"be64e709-b1cb-4287-8bff-e6829a7e3134","Type":"ContainerDied","Data":"bdd5ca02afc1bb9de940833932bba7b1186cc6369df5a8563adbd4048c39479e"} Jul 07 16:34:50.056271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:50.056239 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" event={"ID":"be64e709-b1cb-4287-8bff-e6829a7e3134","Type":"ContainerStarted","Data":"690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c"} Jul 07 16:34:51.063619 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.063597 2569 generic.go:356] "Generic (PLEG): container finished" podID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerID="31154feaae85559fa24d013ea58b7d9897f28ddc56c8eae46bcdfc2da3c733ba" exitCode=0 Jul 07 16:34:51.063975 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.063675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" event={"ID":"f473e35c-7340-4b50-8293-b2004f9f7e89","Type":"ContainerDied","Data":"31154feaae85559fa24d013ea58b7d9897f28ddc56c8eae46bcdfc2da3c733ba"} Jul 07 16:34:51.065480 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.065459 2569 generic.go:356] "Generic (PLEG): container finished" podID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerID="78b6bacde2e2c95d4ed630a6ef475eee89481e3ea3f212352875bebb05c3ce6d" exitCode=0 Jul 07 16:34:51.065587 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.065544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" event={"ID":"f79b6b94-003f-4802-8ab4-8e4fe0c38eae","Type":"ContainerDied","Data":"78b6bacde2e2c95d4ed630a6ef475eee89481e3ea3f212352875bebb05c3ce6d"} Jul 07 16:34:51.067785 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.067763 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2dcac0f-c709-4603-884d-617f73832558" containerID="7ba764f96d7d76610b1bb91194e8a426bff88761d38b7b695463f4aabcd6e909" exitCode=0 Jul 07 16:34:51.067911 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.067829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" event={"ID":"f2dcac0f-c709-4603-884d-617f73832558","Type":"ContainerDied","Data":"7ba764f96d7d76610b1bb91194e8a426bff88761d38b7b695463f4aabcd6e909"} Jul 07 16:34:51.069560 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.069541 2569 generic.go:356] "Generic (PLEG): container finished" podID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerID="96530b8051f9964f9ce0df13b28fa17d57839e211a8a4a4360fb9ceb8565cf48" exitCode=0 Jul 07 16:34:51.069644 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:51.069563 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" event={"ID":"be64e709-b1cb-4287-8bff-e6829a7e3134","Type":"ContainerDied","Data":"96530b8051f9964f9ce0df13b28fa17d57839e211a8a4a4360fb9ceb8565cf48"} Jul 07 16:34:52.077476 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.077442 2569 generic.go:356] "Generic (PLEG): container finished" podID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerID="48c2a10041d13909dda558a8da58afdab8b51ce5d78f8e98b09ae9a74756cb84" exitCode=0 Jul 07 16:34:52.077908 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.077502 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" event={"ID":"f79b6b94-003f-4802-8ab4-8e4fe0c38eae","Type":"ContainerDied","Data":"48c2a10041d13909dda558a8da58afdab8b51ce5d78f8e98b09ae9a74756cb84"} Jul 07 16:34:52.079882 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.079859 2569 generic.go:356] "Generic (PLEG): container finished" podID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerID="c21d1877917a658f53efe30709781286116a40fd38a97c212bea558f7c2ce442" exitCode=0 Jul 07 16:34:52.080024 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.079920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" event={"ID":"be64e709-b1cb-4287-8bff-e6829a7e3134","Type":"ContainerDied","Data":"c21d1877917a658f53efe30709781286116a40fd38a97c212bea558f7c2ce442"} Jul 07 16:34:52.212940 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.212915 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:52.243770 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.243750 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:52.310943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.310915 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxknx\" (UniqueName: \"kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx\") pod \"f2dcac0f-c709-4603-884d-617f73832558\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " Jul 07 16:34:52.311080 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.310950 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util\") pod \"f2dcac0f-c709-4603-884d-617f73832558\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " Jul 07 16:34:52.311080 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311006 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle\") pod \"f2dcac0f-c709-4603-884d-617f73832558\" (UID: \"f2dcac0f-c709-4603-884d-617f73832558\") " Jul 07 16:34:52.311080 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle\") pod \"f473e35c-7340-4b50-8293-b2004f9f7e89\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " Jul 07 16:34:52.311080 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311066 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lm4m\" (UniqueName: \"kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m\") pod \"f473e35c-7340-4b50-8293-b2004f9f7e89\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " Jul 07 16:34:52.311302 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311106 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util\") pod \"f473e35c-7340-4b50-8293-b2004f9f7e89\" (UID: \"f473e35c-7340-4b50-8293-b2004f9f7e89\") " Jul 07 16:34:52.311573 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311544 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle" (OuterVolumeSpecName: "bundle") pod "f2dcac0f-c709-4603-884d-617f73832558" (UID: "f2dcac0f-c709-4603-884d-617f73832558"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:52.311696 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.311553 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle" (OuterVolumeSpecName: "bundle") pod "f473e35c-7340-4b50-8293-b2004f9f7e89" (UID: "f473e35c-7340-4b50-8293-b2004f9f7e89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:52.312231 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.312205 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:52.312338 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.312235 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:52.313573 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.313542 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx" (OuterVolumeSpecName: "kube-api-access-rxknx") pod "f2dcac0f-c709-4603-884d-617f73832558" (UID: "f2dcac0f-c709-4603-884d-617f73832558"). InnerVolumeSpecName "kube-api-access-rxknx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:34:52.313686 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.313625 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m" (OuterVolumeSpecName: "kube-api-access-7lm4m") pod "f473e35c-7340-4b50-8293-b2004f9f7e89" (UID: "f473e35c-7340-4b50-8293-b2004f9f7e89"). InnerVolumeSpecName "kube-api-access-7lm4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:34:52.316706 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.316678 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util" (OuterVolumeSpecName: "util") pod "f2dcac0f-c709-4603-884d-617f73832558" (UID: "f2dcac0f-c709-4603-884d-617f73832558"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:52.317425 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.317404 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util" (OuterVolumeSpecName: "util") pod "f473e35c-7340-4b50-8293-b2004f9f7e89" (UID: "f473e35c-7340-4b50-8293-b2004f9f7e89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:52.412758 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.412730 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxknx\" (UniqueName: \"kubernetes.io/projected/f2dcac0f-c709-4603-884d-617f73832558-kube-api-access-rxknx\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:52.412758 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.412754 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2dcac0f-c709-4603-884d-617f73832558-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:52.412916 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.412769 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lm4m\" (UniqueName: \"kubernetes.io/projected/f473e35c-7340-4b50-8293-b2004f9f7e89-kube-api-access-7lm4m\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:52.412916 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:52.412781 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f473e35c-7340-4b50-8293-b2004f9f7e89-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.087147 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.087123 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" Jul 07 16:34:53.087516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.087155 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm" event={"ID":"f2dcac0f-c709-4603-884d-617f73832558","Type":"ContainerDied","Data":"c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3"} Jul 07 16:34:53.087516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.087210 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3fb237287e4af781af8d54ed7876bc5f545defabcdea28dbd6f3f2b1b9a8db3" Jul 07 16:34:53.089283 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.089266 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" Jul 07 16:34:53.089399 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.089298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst" event={"ID":"f473e35c-7340-4b50-8293-b2004f9f7e89","Type":"ContainerDied","Data":"0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae"} Jul 07 16:34:53.089399 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.089336 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfb3da3b78b32970ba65bfce7f4ce7b7f998ec1e4a905f02ebfaed1652bc5ae" Jul 07 16:34:53.221767 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.221746 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:53.252194 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.252174 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:53.319011 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.318989 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util\") pod \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " Jul 07 16:34:53.319145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319044 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle\") pod \"be64e709-b1cb-4287-8bff-e6829a7e3134\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " Jul 07 16:34:53.319145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319077 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle\") pod \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " Jul 07 16:34:53.319145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319117 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dh62\" (UniqueName: \"kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62\") pod \"be64e709-b1cb-4287-8bff-e6829a7e3134\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " Jul 07 16:34:53.319295 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319150 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util\") pod \"be64e709-b1cb-4287-8bff-e6829a7e3134\" (UID: \"be64e709-b1cb-4287-8bff-e6829a7e3134\") " Jul 07 16:34:53.319295 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319193 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk6sb\" (UniqueName: \"kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb\") pod \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\" (UID: \"f79b6b94-003f-4802-8ab4-8e4fe0c38eae\") " Jul 07 16:34:53.319733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319707 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle" (OuterVolumeSpecName: "bundle") pod "f79b6b94-003f-4802-8ab4-8e4fe0c38eae" (UID: "f79b6b94-003f-4802-8ab4-8e4fe0c38eae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:53.319828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.319781 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle" (OuterVolumeSpecName: "bundle") pod "be64e709-b1cb-4287-8bff-e6829a7e3134" (UID: "be64e709-b1cb-4287-8bff-e6829a7e3134"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:53.321253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.321234 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62" (OuterVolumeSpecName: "kube-api-access-4dh62") pod "be64e709-b1cb-4287-8bff-e6829a7e3134" (UID: "be64e709-b1cb-4287-8bff-e6829a7e3134"). InnerVolumeSpecName "kube-api-access-4dh62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:34:53.321626 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.321609 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb" (OuterVolumeSpecName: "kube-api-access-xk6sb") pod "f79b6b94-003f-4802-8ab4-8e4fe0c38eae" (UID: "f79b6b94-003f-4802-8ab4-8e4fe0c38eae"). InnerVolumeSpecName "kube-api-access-xk6sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:34:53.325015 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.324981 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util" (OuterVolumeSpecName: "util") pod "be64e709-b1cb-4287-8bff-e6829a7e3134" (UID: "be64e709-b1cb-4287-8bff-e6829a7e3134"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:53.325070 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.324983 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util" (OuterVolumeSpecName: "util") pod "f79b6b94-003f-4802-8ab4-8e4fe0c38eae" (UID: "f79b6b94-003f-4802-8ab4-8e4fe0c38eae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420153 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420186 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420195 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420203 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dh62\" (UniqueName: \"kubernetes.io/projected/be64e709-b1cb-4287-8bff-e6829a7e3134-kube-api-access-4dh62\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420213 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be64e709-b1cb-4287-8bff-e6829a7e3134-util\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:53.420214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:53.420222 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6sb\" (UniqueName: \"kubernetes.io/projected/f79b6b94-003f-4802-8ab4-8e4fe0c38eae-kube-api-access-xk6sb\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:34:54.098465 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.098436 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" Jul 07 16:34:54.098465 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.098448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m" event={"ID":"f79b6b94-003f-4802-8ab4-8e4fe0c38eae","Type":"ContainerDied","Data":"65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1"} Jul 07 16:34:54.098938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.098506 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b8cb5563815b4ad5393f9aa82873a378a35f360a41326638bdfd20bf0de5c1" Jul 07 16:34:54.100874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.100822 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" event={"ID":"be64e709-b1cb-4287-8bff-e6829a7e3134","Type":"ContainerDied","Data":"690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c"} Jul 07 16:34:54.101000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.100881 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp" Jul 07 16:34:54.101000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:34:54.100894 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690ed1efa09bec8b1ba083767cef5d1b453811559facead56de194cf8a54c94c" Jul 07 16:35:01.222759 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:01.222722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:05.371978 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:05.371934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:07.998702 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:07.998597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:09.715475 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:09.715442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:11.103369 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:11.103327 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:12.599209 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:12.599165 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:12.948503 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:12.948392 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:13.163519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:13.163488 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:15.398738 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:15.398694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:17.587025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:17.586991 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:17.686637 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:17.686578 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:19.389095 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:19.389054 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:19.550306 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:19.550255 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:19.647615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:19.647532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:21.217719 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:21.217668 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:21.287146 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:21.287111 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:22.617991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.617951 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-76nng"] Jul 07 16:35:22.618491 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618468 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="util" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618494 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="util" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618507 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618515 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618527 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618537 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618548 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618556 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="extract" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618570 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="util" Jul 07 16:35:22.618576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618579 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="util" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618589 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="util" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618598 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="util" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618616 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="util" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618625 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="util" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618637 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618647 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618657 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="extract" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618665 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="extract" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618680 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618688 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618715 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618723 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618739 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618747 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="pull" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618855 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2dcac0f-c709-4603-884d-617f73832558" containerName="extract" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618871 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79b6b94-003f-4802-8ab4-8e4fe0c38eae" containerName="extract" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618883 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be64e709-b1cb-4287-8bff-e6829a7e3134" containerName="extract" Jul 07 16:35:22.619113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.618891 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f473e35c-7340-4b50-8293-b2004f9f7e89" containerName="extract" Jul 07 16:35:22.622500 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.622479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.625516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.625488 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7cb8t\"" Jul 07 16:35:22.625648 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.625597 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 16:35:22.626552 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.626528 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 16:35:22.626681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.626561 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 16:35:22.626681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.626587 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 16:35:22.634289 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.634266 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-76nng"] Jul 07 16:35:22.637251 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.637215 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:22.743194 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.743166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/42993fa1-f26c-4226-9776-ffc7d9368d16-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.743296 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.743197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42993fa1-f26c-4226-9776-ffc7d9368d16-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.743296 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.743278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ntg\" (UniqueName: \"kubernetes.io/projected/42993fa1-f26c-4226-9776-ffc7d9368d16-kube-api-access-64ntg\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.806632 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.806602 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:22.843566 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.843542 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ntg\" (UniqueName: \"kubernetes.io/projected/42993fa1-f26c-4226-9776-ffc7d9368d16-kube-api-access-64ntg\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.843667 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.843590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/42993fa1-f26c-4226-9776-ffc7d9368d16-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.843667 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.843609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42993fa1-f26c-4226-9776-ffc7d9368d16-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.844253 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.844230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/42993fa1-f26c-4226-9776-ffc7d9368d16-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.846027 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.846003 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/42993fa1-f26c-4226-9776-ffc7d9368d16-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.853071 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.853050 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ntg\" (UniqueName: \"kubernetes.io/projected/42993fa1-f26c-4226-9776-ffc7d9368d16-kube-api-access-64ntg\") pod \"kuadrant-console-plugin-66b5774985-76nng\" (UID: \"42993fa1-f26c-4226-9776-ffc7d9368d16\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:22.932745 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:22.932697 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" Jul 07 16:35:23.071970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:23.071943 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-76nng"] Jul 07 16:35:23.073567 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:35:23.073520 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42993fa1_f26c_4226_9776_ffc7d9368d16.slice/crio-833f8724f232d4e6828a7c4498ce3a88efd3e821a95909a08085e1edefc75201 WatchSource:0}: Error finding container 833f8724f232d4e6828a7c4498ce3a88efd3e821a95909a08085e1edefc75201: Status 404 returned error can't find the container with id 833f8724f232d4e6828a7c4498ce3a88efd3e821a95909a08085e1edefc75201 Jul 07 16:35:23.253273 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:23.253210 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" event={"ID":"42993fa1-f26c-4226-9776-ffc7d9368d16","Type":"ContainerStarted","Data":"833f8724f232d4e6828a7c4498ce3a88efd3e821a95909a08085e1edefc75201"} Jul 07 16:35:24.764995 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:24.764959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:24.866598 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:24.866554 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:33.081977 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:33.081914 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:33.187528 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:33.187493 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:35.044201 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:35.044149 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:35.146590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:35.146556 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:35.661590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:35.661557 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:35.765263 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:35.765200 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:38.973084 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:38.973053 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:39.073095 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:39.073064 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:43.418758 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:43.418727 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:43.519592 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:43.519541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:47.392118 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:47.392092 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" event={"ID":"42993fa1-f26c-4226-9776-ffc7d9368d16","Type":"ContainerStarted","Data":"b772b8b09d04d9a00e13014b78295e27dfa1188aca86995341c93c048d2d4063"} Jul 07 16:35:47.421079 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:47.421034 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-76nng" podStartSLOduration=1.271291613 podStartE2EDuration="25.421019227s" totalImagesPullingTime="24.149727614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:35:47.417790652 +0000 UTC m=+751.726629930" watchObservedRunningTime="2026-07-07 16:35:47.421019227 +0000 UTC m=+751.729858505" Jul 07 16:35:53.600605 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:53.600573 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:35:53.700103 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:35:53.700067 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:02.405681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:02.405642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:02.598421 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:02.598390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.012072 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.012041 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:04.015381 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.015352 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.055752 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.055724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:04.055887 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.055851 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.058477 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.058457 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 16:36:04.091097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.091072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrk82\" (UniqueName: \"kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.091245 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.091194 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.111560 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.111538 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:04.119458 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.119435 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.185855 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.185812 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.191938 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.191918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.192024 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.191950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrk82\" (UniqueName: \"kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.192497 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.192480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.203740 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.203713 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrk82\" (UniqueName: \"kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82\") pod \"limitador-limitador-64c94fd64f-n45px\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.365786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.365754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:04.505198 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.505174 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:04.505316 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.505297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.508022 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:04.507983 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b6ae7e_599f_4249_a27d_2309c034f536.slice/crio-df703670c70e6dfcd4d751428d16e2bcffbe59f707007f08dfe42d0f3344ccd6 WatchSource:0}: Error finding container df703670c70e6dfcd4d751428d16e2bcffbe59f707007f08dfe42d0f3344ccd6: Status 404 returned error can't find the container with id df703670c70e6dfcd4d751428d16e2bcffbe59f707007f08dfe42d0f3344ccd6 Jul 07 16:36:04.707449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.707388 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.797415 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.797392 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:04.960975 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.960918 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:04.964175 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.964160 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:04.967320 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.967291 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r95wp\"" Jul 07 16:36:04.975763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:04.975744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:05.020640 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.020615 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:05.099078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.099058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpmf\" (UniqueName: \"kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf\") pod \"authorino-58cd955675-gmpsk\" (UID: \"e508e2f2-40ef-4c8c-8028-aa40ab038bf4\") " pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:05.143587 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.143556 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:05.199450 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.199427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpmf\" (UniqueName: \"kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf\") pod \"authorino-58cd955675-gmpsk\" (UID: \"e508e2f2-40ef-4c8c-8028-aa40ab038bf4\") " pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:05.213567 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.213475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpmf\" (UniqueName: \"kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf\") pod \"authorino-58cd955675-gmpsk\" (UID: \"e508e2f2-40ef-4c8c-8028-aa40ab038bf4\") " pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:05.273790 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.273771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:05.451526 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.451496 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:05.453737 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:05.453707 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode508e2f2_40ef_4c8c_8028_aa40ab038bf4.slice/crio-66cd5742c8384eac34abb6b253fa191ff104313c0a33cc68b82c646d2ffbcab3 WatchSource:0}: Error finding container 66cd5742c8384eac34abb6b253fa191ff104313c0a33cc68b82c646d2ffbcab3: Status 404 returned error can't find the container with id 66cd5742c8384eac34abb6b253fa191ff104313c0a33cc68b82c646d2ffbcab3 Jul 07 16:36:05.500149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.500074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gmpsk" event={"ID":"e508e2f2-40ef-4c8c-8028-aa40ab038bf4","Type":"ContainerStarted","Data":"66cd5742c8384eac34abb6b253fa191ff104313c0a33cc68b82c646d2ffbcab3"} Jul 07 16:36:05.502065 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:05.502003 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" event={"ID":"92b6ae7e-599f-4249-a27d-2309c034f536","Type":"ContainerStarted","Data":"df703670c70e6dfcd4d751428d16e2bcffbe59f707007f08dfe42d0f3344ccd6"} Jul 07 16:36:08.963254 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:08.963219 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:09.530005 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.529966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gmpsk" event={"ID":"e508e2f2-40ef-4c8c-8028-aa40ab038bf4","Type":"ContainerStarted","Data":"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4"} Jul 07 16:36:09.530184 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.529996 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-gmpsk" podUID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" containerName="authorino" containerID="cri-o://aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4" gracePeriod=30 Jul 07 16:36:09.531462 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.531441 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" event={"ID":"92b6ae7e-599f-4249-a27d-2309c034f536","Type":"ContainerStarted","Data":"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec"} Jul 07 16:36:09.531585 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.531565 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:09.549788 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.549751 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-gmpsk" podStartSLOduration=1.945011958 podStartE2EDuration="5.549739158s" totalImagesPullingTime="3.6047272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:09.548231609 +0000 UTC m=+773.857070887" watchObservedRunningTime="2026-07-07 16:36:09.549739158 +0000 UTC m=+773.858578436" Jul 07 16:36:09.567157 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.567120 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" podStartSLOduration=2.074369357 podStartE2EDuration="6.567107774s" totalImagesPullingTime="4.492738417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:09.56618688 +0000 UTC m=+773.875026158" watchObservedRunningTime="2026-07-07 16:36:09.567107774 +0000 UTC m=+773.875947052" Jul 07 16:36:09.781153 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.781097 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:09.844863 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.844820 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdpmf\" (UniqueName: \"kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf\") pod \"e508e2f2-40ef-4c8c-8028-aa40ab038bf4\" (UID: \"e508e2f2-40ef-4c8c-8028-aa40ab038bf4\") " Jul 07 16:36:09.846747 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.846718 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf" (OuterVolumeSpecName: "kube-api-access-zdpmf") pod "e508e2f2-40ef-4c8c-8028-aa40ab038bf4" (UID: "e508e2f2-40ef-4c8c-8028-aa40ab038bf4"). InnerVolumeSpecName "kube-api-access-zdpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:36:09.946228 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:09.946205 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpmf\" (UniqueName: \"kubernetes.io/projected/e508e2f2-40ef-4c8c-8028-aa40ab038bf4-kube-api-access-zdpmf\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:36:10.538874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.538779 2569 generic.go:356] "Generic (PLEG): container finished" podID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" containerID="aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4" exitCode=0 Jul 07 16:36:10.538874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.538824 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-gmpsk" Jul 07 16:36:10.539327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.538868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gmpsk" event={"ID":"e508e2f2-40ef-4c8c-8028-aa40ab038bf4","Type":"ContainerDied","Data":"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4"} Jul 07 16:36:10.539327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.538914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-gmpsk" event={"ID":"e508e2f2-40ef-4c8c-8028-aa40ab038bf4","Type":"ContainerDied","Data":"66cd5742c8384eac34abb6b253fa191ff104313c0a33cc68b82c646d2ffbcab3"} Jul 07 16:36:10.539327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.538929 2569 scope.go:117] "RemoveContainer" containerID="aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4" Jul 07 16:36:10.547996 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.547971 2569 scope.go:117] "RemoveContainer" containerID="aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4" Jul 07 16:36:10.548283 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:10.548263 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4\": container with ID starting with aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4 not found: ID does not exist" containerID="aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4" Jul 07 16:36:10.548349 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.548291 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4"} err="failed to get container status \"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4\": rpc error: code = NotFound desc = could not find container \"aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4\": container with ID starting with aca3bf8eb2aeec7dcec82126d66b9ecd4d0d376e5dbdc85f40f015b1361912b4 not found: ID does not exist" Jul 07 16:36:10.588571 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.588550 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:10.592198 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:10.592178 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-gmpsk"] Jul 07 16:36:12.267580 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:12.267538 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" path="/var/lib/kubelet/pods/e508e2f2-40ef-4c8c-8028-aa40ab038bf4/volumes" Jul 07 16:36:19.449203 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:19.449153 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:19.449605 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:19.449377 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" podUID="92b6ae7e-599f-4249-a27d-2309c034f536" containerName="limitador" containerID="cri-o://807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec" gracePeriod=30 Jul 07 16:36:19.450183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:19.450164 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:20.062576 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.062550 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:20.119274 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.119248 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrk82\" (UniqueName: \"kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82\") pod \"92b6ae7e-599f-4249-a27d-2309c034f536\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " Jul 07 16:36:20.119429 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.119309 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file\") pod \"92b6ae7e-599f-4249-a27d-2309c034f536\" (UID: \"92b6ae7e-599f-4249-a27d-2309c034f536\") " Jul 07 16:36:20.119722 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.119696 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file" (OuterVolumeSpecName: "config-file") pod "92b6ae7e-599f-4249-a27d-2309c034f536" (UID: "92b6ae7e-599f-4249-a27d-2309c034f536"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:36:20.121263 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.121239 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82" (OuterVolumeSpecName: "kube-api-access-hrk82") pod "92b6ae7e-599f-4249-a27d-2309c034f536" (UID: "92b6ae7e-599f-4249-a27d-2309c034f536"). InnerVolumeSpecName "kube-api-access-hrk82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:36:20.220480 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.220458 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrk82\" (UniqueName: \"kubernetes.io/projected/92b6ae7e-599f-4249-a27d-2309c034f536-kube-api-access-hrk82\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:36:20.220566 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.220500 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/92b6ae7e-599f-4249-a27d-2309c034f536-config-file\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:36:20.598360 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.598331 2569 generic.go:356] "Generic (PLEG): container finished" podID="92b6ae7e-599f-4249-a27d-2309c034f536" containerID="807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec" exitCode=0 Jul 07 16:36:20.598694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.598380 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" Jul 07 16:36:20.598694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.598420 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" event={"ID":"92b6ae7e-599f-4249-a27d-2309c034f536","Type":"ContainerDied","Data":"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec"} Jul 07 16:36:20.598694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.598480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n45px" event={"ID":"92b6ae7e-599f-4249-a27d-2309c034f536","Type":"ContainerDied","Data":"df703670c70e6dfcd4d751428d16e2bcffbe59f707007f08dfe42d0f3344ccd6"} Jul 07 16:36:20.598694 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.598510 2569 scope.go:117] "RemoveContainer" containerID="807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec" Jul 07 16:36:20.608030 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.608015 2569 scope.go:117] "RemoveContainer" containerID="807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec" Jul 07 16:36:20.608304 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:20.608279 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec\": container with ID starting with 807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec not found: ID does not exist" containerID="807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec" Jul 07 16:36:20.608421 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.608309 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec"} err="failed to get container status \"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec\": rpc error: code = NotFound desc = could not find container \"807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec\": container with ID starting with 807e423105e032e1da1e697301b5afc01e74fe171c1d09170f58509ce183b9ec not found: ID does not exist" Jul 07 16:36:20.617804 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.617779 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:20.621597 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:20.621576 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n45px"] Jul 07 16:36:22.267720 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:22.267679 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b6ae7e-599f-4249-a27d-2309c034f536" path="/var/lib/kubelet/pods/92b6ae7e-599f-4249-a27d-2309c034f536/volumes" Jul 07 16:36:25.744661 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.744628 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-n97dk"] Jul 07 16:36:25.745183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745166 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b6ae7e-599f-4249-a27d-2309c034f536" containerName="limitador" Jul 07 16:36:25.745234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745186 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b6ae7e-599f-4249-a27d-2309c034f536" containerName="limitador" Jul 07 16:36:25.745234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745201 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" containerName="authorino" Jul 07 16:36:25.745234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745209 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" containerName="authorino" Jul 07 16:36:25.745327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745306 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b6ae7e-599f-4249-a27d-2309c034f536" containerName="limitador" Jul 07 16:36:25.745327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.745321 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e508e2f2-40ef-4c8c-8028-aa40ab038bf4" containerName="authorino" Jul 07 16:36:25.749888 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.749870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.753345 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.753329 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mbkx6\"" Jul 07 16:36:25.753788 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.753775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 07 16:36:25.761782 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.761762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-n97dk"] Jul 07 16:36:25.794849 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.794812 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:25.862079 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.862058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrrp\" (UniqueName: \"kubernetes.io/projected/1d973308-f677-4de8-9a4d-de98aeefd85d-kube-api-access-5xrrp\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.862184 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.862122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d973308-f677-4de8-9a4d-de98aeefd85d-data\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.962836 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.962815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d973308-f677-4de8-9a4d-de98aeefd85d-data\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.962925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.962876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrrp\" (UniqueName: \"kubernetes.io/projected/1d973308-f677-4de8-9a4d-de98aeefd85d-kube-api-access-5xrrp\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.963165 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.963148 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d973308-f677-4de8-9a4d-de98aeefd85d-data\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:25.976627 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:25.976599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrrp\" (UniqueName: \"kubernetes.io/projected/1d973308-f677-4de8-9a4d-de98aeefd85d-kube-api-access-5xrrp\") pod \"postgres-69fb4594cf-n97dk\" (UID: \"1d973308-f677-4de8-9a4d-de98aeefd85d\") " pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:26.060015 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:26.059964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:26.198344 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:26.198309 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-n97dk"] Jul 07 16:36:26.201352 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:26.201269 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d973308_f677_4de8_9a4d_de98aeefd85d.slice/crio-002d4e12115bc8567aca3b1839cf3c7c0e2ab1a96b5ab21d1040c47100dca9df WatchSource:0}: Error finding container 002d4e12115bc8567aca3b1839cf3c7c0e2ab1a96b5ab21d1040c47100dca9df: Status 404 returned error can't find the container with id 002d4e12115bc8567aca3b1839cf3c7c0e2ab1a96b5ab21d1040c47100dca9df Jul 07 16:36:26.633439 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:26.633397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-n97dk" event={"ID":"1d973308-f677-4de8-9a4d-de98aeefd85d","Type":"ContainerStarted","Data":"002d4e12115bc8567aca3b1839cf3c7c0e2ab1a96b5ab21d1040c47100dca9df"} Jul 07 16:36:31.668502 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:31.668466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-n97dk" event={"ID":"1d973308-f677-4de8-9a4d-de98aeefd85d","Type":"ContainerStarted","Data":"16d96e2652b5aa067852d9b81702d4d6977be4dda08ca16ddfb575bd32bbcfff"} Jul 07 16:36:31.668860 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:31.668556 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:31.692241 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:31.692160 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-n97dk" podStartSLOduration=1.4687811530000001 podStartE2EDuration="6.692144898s" totalImagesPullingTime="5.223363745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:31.690101248 +0000 UTC m=+795.998940526" watchObservedRunningTime="2026-07-07 16:36:31.692144898 +0000 UTC m=+796.000984175" Jul 07 16:36:37.704971 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:37.704937 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-n97dk" Jul 07 16:36:38.063514 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.063422 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:38.164275 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.164224 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:38.344145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.344119 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4zrdv"] Jul 07 16:36:38.397540 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.397514 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4zrdv"] Jul 07 16:36:38.397645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.397588 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.400907 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.400888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-r95wp\"" Jul 07 16:36:38.563092 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.563063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbd5\" (UniqueName: \"kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5\") pod \"authorino-5d4b67c446-4zrdv\" (UID: \"8d233033-2bc4-4e5c-b55e-4a39f6ce1c19\") " pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.633394 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.633335 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4zrdv"] Jul 07 16:36:38.633538 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:38.633519 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8rbd5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-4zrdv" podUID="8d233033-2bc4-4e5c-b55e-4a39f6ce1c19" Jul 07 16:36:38.664255 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.664227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbd5\" (UniqueName: \"kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5\") pod \"authorino-5d4b67c446-4zrdv\" (UID: \"8d233033-2bc4-4e5c-b55e-4a39f6ce1c19\") " pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.704515 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.704493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.707943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.707920 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-666db9788-jrhft"] Jul 07 16:36:38.708487 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.708463 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbd5\" (UniqueName: \"kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5\") pod \"authorino-5d4b67c446-4zrdv\" (UID: \"8d233033-2bc4-4e5c-b55e-4a39f6ce1c19\") " pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.743178 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.743157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-666db9788-jrhft"] Jul 07 16:36:38.743296 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.743246 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:38.745119 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.745095 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:38.854176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.854145 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-666db9788-jrhft"] Jul 07 16:36:38.854391 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:38.854370 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rshsm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-666db9788-jrhft" podUID="76e95d0c-3c1c-47b4-be82-1cd23f8d561b" Jul 07 16:36:38.865916 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.865891 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rbd5\" (UniqueName: \"kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5\") pod \"8d233033-2bc4-4e5c-b55e-4a39f6ce1c19\" (UID: \"8d233033-2bc4-4e5c-b55e-4a39f6ce1c19\") " Jul 07 16:36:38.866086 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.866070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshsm\" (UniqueName: \"kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm\") pod \"authorino-666db9788-jrhft\" (UID: \"76e95d0c-3c1c-47b4-be82-1cd23f8d561b\") " pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:38.867651 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.867625 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5" (OuterVolumeSpecName: "kube-api-access-8rbd5") pod "8d233033-2bc4-4e5c-b55e-4a39f6ce1c19" (UID: "8d233033-2bc4-4e5c-b55e-4a39f6ce1c19"). InnerVolumeSpecName "kube-api-access-8rbd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:36:38.895112 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.895066 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:36:38.931786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.931764 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:36:38.931903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.931862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:38.934469 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.934453 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 16:36:38.966706 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.966681 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshsm\" (UniqueName: \"kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm\") pod \"authorino-666db9788-jrhft\" (UID: \"76e95d0c-3c1c-47b4-be82-1cd23f8d561b\") " pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:38.966794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.966724 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rbd5\" (UniqueName: \"kubernetes.io/projected/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19-kube-api-access-8rbd5\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:36:38.975235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:38.975211 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshsm\" (UniqueName: \"kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm\") pod \"authorino-666db9788-jrhft\" (UID: \"76e95d0c-3c1c-47b4-be82-1cd23f8d561b\") " pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:39.067120 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.067092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9n6\" (UniqueName: \"kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.067233 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.067156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.168078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.168015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9n6\" (UniqueName: \"kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.168180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.168074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.170310 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.170291 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.176470 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.176442 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9n6\" (UniqueName: \"kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6\") pod \"authorino-6dc887ffcb-rtvb8\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.240872 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.240850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:36:39.403728 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.403645 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:36:39.406491 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:39.406465 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod365e4722_bc8a_4e6a_b08d_25ffb2caede9.slice/crio-29def7103b3e1b38a6dcd6e517cba1da620e6b752aebea8e635d46f8f3418cc1 WatchSource:0}: Error finding container 29def7103b3e1b38a6dcd6e517cba1da620e6b752aebea8e635d46f8f3418cc1: Status 404 returned error can't find the container with id 29def7103b3e1b38a6dcd6e517cba1da620e6b752aebea8e635d46f8f3418cc1 Jul 07 16:36:39.712308 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.712283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:39.712693 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.712308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" event={"ID":"365e4722-bc8a-4e6a-b08d-25ffb2caede9","Type":"ContainerStarted","Data":"29def7103b3e1b38a6dcd6e517cba1da620e6b752aebea8e635d46f8f3418cc1"} Jul 07 16:36:39.712693 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.712428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4zrdv" Jul 07 16:36:39.756605 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.756584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:39.779179 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.779157 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4zrdv"] Jul 07 16:36:39.783673 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.783652 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4zrdv"] Jul 07 16:36:39.875052 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.875027 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rshsm\" (UniqueName: \"kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm\") pod \"76e95d0c-3c1c-47b4-be82-1cd23f8d561b\" (UID: \"76e95d0c-3c1c-47b4-be82-1cd23f8d561b\") " Jul 07 16:36:39.877042 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.877012 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm" (OuterVolumeSpecName: "kube-api-access-rshsm") pod "76e95d0c-3c1c-47b4-be82-1cd23f8d561b" (UID: "76e95d0c-3c1c-47b4-be82-1cd23f8d561b"). InnerVolumeSpecName "kube-api-access-rshsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:36:39.976000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:39.975938 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshsm\" (UniqueName: \"kubernetes.io/projected/76e95d0c-3c1c-47b4-be82-1cd23f8d561b-kube-api-access-rshsm\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:36:40.269010 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.268932 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d233033-2bc4-4e5c-b55e-4a39f6ce1c19" path="/var/lib/kubelet/pods/8d233033-2bc4-4e5c-b55e-4a39f6ce1c19/volumes" Jul 07 16:36:40.719489 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.719452 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" event={"ID":"365e4722-bc8a-4e6a-b08d-25ffb2caede9","Type":"ContainerStarted","Data":"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f"} Jul 07 16:36:40.719952 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.719521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666db9788-jrhft" Jul 07 16:36:40.742735 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.742689 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" podStartSLOduration=2.092498357 podStartE2EDuration="2.742675761s" totalImagesPullingTime="650.177404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:40.739495738 +0000 UTC m=+805.048335016" watchObservedRunningTime="2026-07-07 16:36:40.742675761 +0000 UTC m=+805.051515038" Jul 07 16:36:40.788674 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.788639 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-666db9788-jrhft"] Jul 07 16:36:40.829759 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:40.829733 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-666db9788-jrhft"] Jul 07 16:36:42.267428 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:42.267396 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e95d0c-3c1c-47b4-be82-1cd23f8d561b" path="/var/lib/kubelet/pods/76e95d0c-3c1c-47b4-be82-1cd23f8d561b/volumes" Jul 07 16:36:45.620036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.620003 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:45.637801 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.637773 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:36:45.640471 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.640456 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.655487 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.655463 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sbxp6\"" Jul 07 16:36:45.668481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.668455 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 16:36:45.668481 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.668460 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 16:36:45.685802 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.685777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:36:45.820232 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.820208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxdl\" (UniqueName: \"kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.820345 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.820288 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.820345 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.820338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.852604 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.852580 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:45.921650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.921599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.921650 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.921637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.921756 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.921669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxdl\" (UniqueName: \"kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.922275 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:45.922218 2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 07 16:36:45.924870 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:36:45.922432 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert podName:5d87fd67-b339-4c49-8844-00173c2c9597 nodeName:}" failed. No retries permitted until 2026-07-07 16:36:46.422408664 +0000 UTC m=+810.731247932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert") pod "maas-controller-6584467cd5-sn7dg" (UID: "5d87fd67-b339-4c49-8844-00173c2c9597") : secret "maas-controller-webhook-cert" not found Jul 07 16:36:45.924870 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.923141 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.930165 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.927579 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:36:45.931548 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.931530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:45.959096 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.959072 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxdl\" (UniqueName: \"kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:45.964944 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:45.964921 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:36:46.029943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.029905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:46.123818 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.123794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7bj\" (UniqueName: \"kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.123925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.123897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.123925 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.123917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.129177 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.129158 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:46.225112 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.225058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.225112 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.225084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.225237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.225142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7bj\" (UniqueName: \"kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.225636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.225617 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.227462 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.227446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.238198 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.238178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7bj\" (UniqueName: \"kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj\") pod \"maas-controller-56dc7d6c67-975nh\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.240954 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.240935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:46.385352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.385326 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:36:46.386551 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:46.386530 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72cca08_0381_40dc_a420_e5adeaa1a33e.slice/crio-8409b18e4ebbc2b14168a0483ad1d03fe020577538929af7483560e0d11de269 WatchSource:0}: Error finding container 8409b18e4ebbc2b14168a0483ad1d03fe020577538929af7483560e0d11de269: Status 404 returned error can't find the container with id 8409b18e4ebbc2b14168a0483ad1d03fe020577538929af7483560e0d11de269 Jul 07 16:36:46.427612 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.427588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:46.429717 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.429695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") pod \"maas-controller-6584467cd5-sn7dg\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:46.550524 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.550472 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:46.683672 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.683643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:36:46.684858 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:36:46.684818 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d87fd67_b339_4c49_8844_00173c2c9597.slice/crio-8cae556f521fec57f2edc8119b3fc46e56d0d83ad4f36247b78ccd66cbccf31f WatchSource:0}: Error finding container 8cae556f521fec57f2edc8119b3fc46e56d0d83ad4f36247b78ccd66cbccf31f: Status 404 returned error can't find the container with id 8cae556f521fec57f2edc8119b3fc46e56d0d83ad4f36247b78ccd66cbccf31f Jul 07 16:36:46.752826 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.752797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6584467cd5-sn7dg" event={"ID":"5d87fd67-b339-4c49-8844-00173c2c9597","Type":"ContainerStarted","Data":"8cae556f521fec57f2edc8119b3fc46e56d0d83ad4f36247b78ccd66cbccf31f"} Jul 07 16:36:46.753978 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:46.753953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-975nh" event={"ID":"b72cca08-0381-40dc-a420-e5adeaa1a33e","Type":"ContainerStarted","Data":"8409b18e4ebbc2b14168a0483ad1d03fe020577538929af7483560e0d11de269"} Jul 07 16:36:48.770343 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.770268 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6584467cd5-sn7dg" event={"ID":"5d87fd67-b339-4c49-8844-00173c2c9597","Type":"ContainerStarted","Data":"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff"} Jul 07 16:36:48.770343 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.770336 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:48.771878 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.771854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-975nh" event={"ID":"b72cca08-0381-40dc-a420-e5adeaa1a33e","Type":"ContainerStarted","Data":"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4"} Jul 07 16:36:48.772034 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.772007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:48.795945 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.795923 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:48.809197 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.809163 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6584467cd5-sn7dg" podStartSLOduration=1.9683405459999999 podStartE2EDuration="3.809148799s" totalImagesPullingTime="1.840808253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:48.80645423 +0000 UTC m=+813.115293508" watchObservedRunningTime="2026-07-07 16:36:48.809148799 +0000 UTC m=+813.117988077" Jul 07 16:36:48.830969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.830925 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56dc7d6c67-975nh" podStartSLOduration=1.7013856779999998 podStartE2EDuration="3.830909601s" totalImagesPullingTime="2.129523923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:36:48.830373155 +0000 UTC m=+813.139212434" watchObservedRunningTime="2026-07-07 16:36:48.830909601 +0000 UTC m=+813.139748882" Jul 07 16:36:48.885922 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:48.885888 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:36:59.783369 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:59.783336 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:36:59.783779 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:59.783469 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:36:59.856388 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:59.856351 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:36:59.856596 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:36:59.856549 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6584467cd5-sn7dg" podUID="5d87fd67-b339-4c49-8844-00173c2c9597" containerName="manager" containerID="cri-o://ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff" gracePeriod=30 Jul 07 16:37:00.100790 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.100765 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:37:00.144773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.144080 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdxdl\" (UniqueName: \"kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl\") pod \"5d87fd67-b339-4c49-8844-00173c2c9597\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " Jul 07 16:37:00.144773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.144138 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca\") pod \"5d87fd67-b339-4c49-8844-00173c2c9597\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " Jul 07 16:37:00.144773 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.144262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") pod \"5d87fd67-b339-4c49-8844-00173c2c9597\" (UID: \"5d87fd67-b339-4c49-8844-00173c2c9597\") " Jul 07 16:37:00.147097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.147060 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d87fd67-b339-4c49-8844-00173c2c9597" (UID: "5d87fd67-b339-4c49-8844-00173c2c9597"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:37:00.147636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.147606 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5d87fd67-b339-4c49-8844-00173c2c9597" (UID: "5d87fd67-b339-4c49-8844-00173c2c9597"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:37:00.149046 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.149019 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl" (OuterVolumeSpecName: "kube-api-access-pdxdl") pod "5d87fd67-b339-4c49-8844-00173c2c9597" (UID: "5d87fd67-b339-4c49-8844-00173c2c9597"). InnerVolumeSpecName "kube-api-access-pdxdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:37:00.245551 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.245531 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d87fd67-b339-4c49-8844-00173c2c9597-webhook-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:37:00.245551 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.245551 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdxdl\" (UniqueName: \"kubernetes.io/projected/5d87fd67-b339-4c49-8844-00173c2c9597-kube-api-access-pdxdl\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:37:00.245671 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.245560 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d87fd67-b339-4c49-8844-00173c2c9597-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:37:00.837649 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.837616 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d87fd67-b339-4c49-8844-00173c2c9597" containerID="ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff" exitCode=0 Jul 07 16:37:00.838036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.837665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6584467cd5-sn7dg" event={"ID":"5d87fd67-b339-4c49-8844-00173c2c9597","Type":"ContainerDied","Data":"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff"} Jul 07 16:37:00.838036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.837674 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6584467cd5-sn7dg" Jul 07 16:37:00.838036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.837731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6584467cd5-sn7dg" event={"ID":"5d87fd67-b339-4c49-8844-00173c2c9597","Type":"ContainerDied","Data":"8cae556f521fec57f2edc8119b3fc46e56d0d83ad4f36247b78ccd66cbccf31f"} Jul 07 16:37:00.838036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.837754 2569 scope.go:117] "RemoveContainer" containerID="ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff" Jul 07 16:37:00.846858 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.846811 2569 scope.go:117] "RemoveContainer" containerID="ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff" Jul 07 16:37:00.847102 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:37:00.847084 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff\": container with ID starting with ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff not found: ID does not exist" containerID="ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff" Jul 07 16:37:00.847159 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.847115 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff"} err="failed to get container status \"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff\": rpc error: code = NotFound desc = could not find container \"ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff\": container with ID starting with ccc17993da54ce706e263f25cc7fb52f7c68e02dd659d91d358863511cc907ff not found: ID does not exist" Jul 07 16:37:00.859422 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.859399 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:37:00.865249 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:00.865230 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6584467cd5-sn7dg"] Jul 07 16:37:02.267480 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:02.267447 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d87fd67-b339-4c49-8844-00173c2c9597" path="/var/lib/kubelet/pods/5d87fd67-b339-4c49-8844-00173c2c9597/volumes" Jul 07 16:37:19.117144 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:19.117110 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:19.644095 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:19.644065 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:20.231881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.231846 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:20.279445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.279412 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-zb5zb"] Jul 07 16:37:20.279763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.279751 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d87fd67-b339-4c49-8844-00173c2c9597" containerName="manager" Jul 07 16:37:20.279811 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.279765 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d87fd67-b339-4c49-8844-00173c2c9597" containerName="manager" Jul 07 16:37:20.279874 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.279864 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d87fd67-b339-4c49-8844-00173c2c9597" containerName="manager" Jul 07 16:37:20.283237 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.283219 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.286082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.286063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 16:37:20.286485 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.286469 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vkr9s\"" Jul 07 16:37:20.308271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.308246 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-zb5zb"] Jul 07 16:37:20.344492 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.344471 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:20.394728 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.394703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnc4\" (UniqueName: \"kubernetes.io/projected/7292ff97-1198-4bce-8f35-2c4da06d79e2-kube-api-access-kgnc4\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.394864 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.394736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7292ff97-1198-4bce-8f35-2c4da06d79e2-plugins-config-volume\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.430254 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.430231 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-84j4f"] Jul 07 16:37:20.433040 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.433023 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.446761 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.446741 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-84j4f"] Jul 07 16:37:20.495262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.495195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fnl\" (UniqueName: \"kubernetes.io/projected/960c0798-0406-4a62-b695-bc3a4fef2020-kube-api-access-j9fnl\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.495262 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.495241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnc4\" (UniqueName: \"kubernetes.io/projected/7292ff97-1198-4bce-8f35-2c4da06d79e2-kube-api-access-kgnc4\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.495445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.495297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7292ff97-1198-4bce-8f35-2c4da06d79e2-plugins-config-volume\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.495445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.495369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960c0798-0406-4a62-b695-bc3a4fef2020-plugins-config-volume\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.496003 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.495983 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7292ff97-1198-4bce-8f35-2c4da06d79e2-plugins-config-volume\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.503698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.503677 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnc4\" (UniqueName: \"kubernetes.io/projected/7292ff97-1198-4bce-8f35-2c4da06d79e2-kube-api-access-kgnc4\") pod \"payload-pre-processing-7964457696-zb5zb\" (UID: \"7292ff97-1198-4bce-8f35-2c4da06d79e2\") " pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.594001 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.593976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:20.595962 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.595940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fnl\" (UniqueName: \"kubernetes.io/projected/960c0798-0406-4a62-b695-bc3a4fef2020-kube-api-access-j9fnl\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.596078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.596009 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960c0798-0406-4a62-b695-bc3a4fef2020-plugins-config-volume\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.596596 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.596578 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960c0798-0406-4a62-b695-bc3a4fef2020-plugins-config-volume\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.606445 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.606425 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fnl\" (UniqueName: \"kubernetes.io/projected/960c0798-0406-4a62-b695-bc3a4fef2020-kube-api-access-j9fnl\") pod \"payload-processing-58f86c7798-84j4f\" (UID: \"960c0798-0406-4a62-b695-bc3a4fef2020\") " pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.729992 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.729964 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-zb5zb"] Jul 07 16:37:20.730856 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:37:20.730809 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7292ff97_1198_4bce_8f35_2c4da06d79e2.slice/crio-e1777af315e7a63905e93aa7bb0eab0ceb38910ef00442368951af91de9e8691 WatchSource:0}: Error finding container e1777af315e7a63905e93aa7bb0eab0ceb38910ef00442368951af91de9e8691: Status 404 returned error can't find the container with id e1777af315e7a63905e93aa7bb0eab0ceb38910ef00442368951af91de9e8691 Jul 07 16:37:20.743026 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.743005 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:20.880167 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.880140 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-84j4f"] Jul 07 16:37:20.881458 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:37:20.881433 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960c0798_0406_4a62_b695_bc3a4fef2020.slice/crio-11507c035cfa3bd9b504ea846e33df00aad7ac4183bd0690026f295ff4f86165 WatchSource:0}: Error finding container 11507c035cfa3bd9b504ea846e33df00aad7ac4183bd0690026f295ff4f86165: Status 404 returned error can't find the container with id 11507c035cfa3bd9b504ea846e33df00aad7ac4183bd0690026f295ff4f86165 Jul 07 16:37:20.962648 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.962611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" event={"ID":"960c0798-0406-4a62-b695-bc3a4fef2020","Type":"ContainerStarted","Data":"11507c035cfa3bd9b504ea846e33df00aad7ac4183bd0690026f295ff4f86165"} Jul 07 16:37:20.963792 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:20.963769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" event={"ID":"7292ff97-1198-4bce-8f35-2c4da06d79e2","Type":"ContainerStarted","Data":"e1777af315e7a63905e93aa7bb0eab0ceb38910ef00442368951af91de9e8691"} Jul 07 16:37:22.975726 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:22.975694 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" event={"ID":"960c0798-0406-4a62-b695-bc3a4fef2020","Type":"ContainerStarted","Data":"45b1be7fd4f0b2759be31fb7d788ff311bcdfd7286c47f549b17e60a53d629a3"} Jul 07 16:37:22.976214 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:22.975879 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:22.977287 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:22.977264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" event={"ID":"7292ff97-1198-4bce-8f35-2c4da06d79e2","Type":"ContainerStarted","Data":"2afc5708cb44769bb309c772eefd39b2a332160aea7e6fb02bcb28f5d944cc77"} Jul 07 16:37:22.977379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:22.977314 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:23.001945 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:23.001891 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" podStartSLOduration=1.508567462 podStartE2EDuration="3.001879195s" totalImagesPullingTime="1.493311733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:37:23.001002346 +0000 UTC m=+847.309841626" watchObservedRunningTime="2026-07-07 16:37:23.001879195 +0000 UTC m=+847.310718472" Jul 07 16:37:23.024419 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:23.024382 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" podStartSLOduration=1.3779185059999999 podStartE2EDuration="3.02436684s" totalImagesPullingTime="1.646448334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:37:23.023518265 +0000 UTC m=+847.332357543" watchObservedRunningTime="2026-07-07 16:37:23.02436684 +0000 UTC m=+847.333206120" Jul 07 16:37:30.278888 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:30.278853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:30.389680 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:30.389640 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:33.984949 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:33.984920 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-84j4f" Jul 07 16:37:33.985368 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:33.984982 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-zb5zb" Jul 07 16:37:39.654559 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:39.654512 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:37:44.951171 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:37:44.951136 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:38:00.084354 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.084315 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l"] Jul 07 16:38:00.089711 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.089692 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.092967 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.092949 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 16:38:00.093969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.093952 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwbt\"" Jul 07 16:38:00.094062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.093994 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 07 16:38:00.094062 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.094054 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 16:38:00.100172 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.100151 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l"] Jul 07 16:38:00.186738 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.186712 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:38:00.201760 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201727 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec10fb-621e-4317-82a7-024d69e9ef00-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.201883 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.201883 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201814 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.201883 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.201994 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201916 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.202029 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.201997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrj4\" (UniqueName: \"kubernetes.io/projected/c2ec10fb-621e-4317-82a7-024d69e9ef00-kube-api-access-qfrj4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303098 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrj4\" (UniqueName: \"kubernetes.io/projected/c2ec10fb-621e-4317-82a7-024d69e9ef00-kube-api-access-qfrj4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec10fb-621e-4317-82a7-024d69e9ef00-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303137 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303235 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303321 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303595 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303664 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303632 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.303664 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.303656 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.305344 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.305324 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2ec10fb-621e-4317-82a7-024d69e9ef00-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.305613 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.305592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2ec10fb-621e-4317-82a7-024d69e9ef00-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.311197 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.311176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrj4\" (UniqueName: \"kubernetes.io/projected/c2ec10fb-621e-4317-82a7-024d69e9ef00-kube-api-access-qfrj4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qvg6l\" (UID: \"c2ec10fb-621e-4317-82a7-024d69e9ef00\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.400865 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.400792 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:00.540613 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:00.540586 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l"] Jul 07 16:38:00.542273 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:38:00.542246 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec10fb_621e_4317_82a7_024d69e9ef00.slice/crio-9747e5ca410a08b0a1bdb260ed4408a83914c9a2e77515e06914af1dc8073e95 WatchSource:0}: Error finding container 9747e5ca410a08b0a1bdb260ed4408a83914c9a2e77515e06914af1dc8073e95: Status 404 returned error can't find the container with id 9747e5ca410a08b0a1bdb260ed4408a83914c9a2e77515e06914af1dc8073e95 Jul 07 16:38:01.216428 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:01.215504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" event={"ID":"c2ec10fb-621e-4317-82a7-024d69e9ef00","Type":"ContainerStarted","Data":"9747e5ca410a08b0a1bdb260ed4408a83914c9a2e77515e06914af1dc8073e95"} Jul 07 16:38:02.224222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:02.224180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" event={"ID":"c2ec10fb-621e-4317-82a7-024d69e9ef00","Type":"ContainerStarted","Data":"5abe74f3ce4cdabfa7cdb53319f9573a96e25339b87e9df00b6eeec0c81ccafd"} Jul 07 16:38:02.250252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:02.250198 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" podStartSLOduration=1.106125271 podStartE2EDuration="2.250181905s" totalImagesPullingTime="1.144056634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:38:02.248926538 +0000 UTC m=+886.557765817" watchObservedRunningTime="2026-07-07 16:38:02.250181905 +0000 UTC m=+886.559021183" Jul 07 16:38:04.127974 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:04.127940 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:38:07.516652 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:07.516616 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:38:10.401978 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:10.401892 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:10.401978 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:10.401952 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:10.414553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:10.414525 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:11.285016 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:11.284983 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qvg6l" Jul 07 16:38:16.163763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:16.163736 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:38:16.166520 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:16.166499 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:38:17.113121 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:38:17.113091 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:39:12.234037 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.234002 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d5597fcd5-c472h"] Jul 07 16:39:12.237791 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.237768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.252542 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.252517 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d5597fcd5-c472h"] Jul 07 16:39:12.348962 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.348935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-tls-cert\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.349074 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.348992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk76d\" (UniqueName: \"kubernetes.io/projected/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-kube-api-access-lk76d\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.450143 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.450116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk76d\" (UniqueName: \"kubernetes.io/projected/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-kube-api-access-lk76d\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.450234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.450179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-tls-cert\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.452623 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.452595 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-tls-cert\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.459435 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.459408 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk76d\" (UniqueName: \"kubernetes.io/projected/d2ee68c1-4d2a-4985-b6b8-c9960946ad7d-kube-api-access-lk76d\") pod \"authorino-5d5597fcd5-c472h\" (UID: \"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d\") " pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.546496 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.546445 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d5597fcd5-c472h" Jul 07 16:39:12.674350 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.674318 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:39:12.706703 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.706675 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d5597fcd5-c472h"] Jul 07 16:39:12.710717 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:39:12.710687 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2ee68c1_4d2a_4985_b6b8_c9960946ad7d.slice/crio-68569cc1365e19d4e5cd0be71805c9b15c906a3ff0766ab193ff39817619699a WatchSource:0}: Error finding container 68569cc1365e19d4e5cd0be71805c9b15c906a3ff0766ab193ff39817619699a: Status 404 returned error can't find the container with id 68569cc1365e19d4e5cd0be71805c9b15c906a3ff0766ab193ff39817619699a Jul 07 16:39:12.711957 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.711938 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:39:12.834054 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:12.834030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:39:13.611343 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.611297 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d5597fcd5-c472h" event={"ID":"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d","Type":"ContainerStarted","Data":"fdef7184f1d4b4bb82126140899f928429c427d1bb8123f711ab1fcd63567113"} Jul 07 16:39:13.611864 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.611352 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d5597fcd5-c472h" event={"ID":"d2ee68c1-4d2a-4985-b6b8-c9960946ad7d","Type":"ContainerStarted","Data":"68569cc1365e19d4e5cd0be71805c9b15c906a3ff0766ab193ff39817619699a"} Jul 07 16:39:13.642082 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.642032 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d5597fcd5-c472h" podStartSLOduration=1.218099282 podStartE2EDuration="1.64201415s" totalImagesPullingTime="423.914868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:39:13.639114929 +0000 UTC m=+957.947954202" watchObservedRunningTime="2026-07-07 16:39:13.64201415 +0000 UTC m=+957.950853427" Jul 07 16:39:13.701146 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.701108 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:39:13.701320 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.701298 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" podUID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" containerName="authorino" containerID="cri-o://42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f" gracePeriod=30 Jul 07 16:39:13.948013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.947973 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:39:13.966165 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:13.966140 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:39:14.064568 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.064539 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v9n6\" (UniqueName: \"kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6\") pod \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " Jul 07 16:39:14.064726 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.064667 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert\") pod \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\" (UID: \"365e4722-bc8a-4e6a-b08d-25ffb2caede9\") " Jul 07 16:39:14.066755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.066710 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6" (OuterVolumeSpecName: "kube-api-access-8v9n6") pod "365e4722-bc8a-4e6a-b08d-25ffb2caede9" (UID: "365e4722-bc8a-4e6a-b08d-25ffb2caede9"). InnerVolumeSpecName "kube-api-access-8v9n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:39:14.076717 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.076687 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "365e4722-bc8a-4e6a-b08d-25ffb2caede9" (UID: "365e4722-bc8a-4e6a-b08d-25ffb2caede9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:39:14.102167 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.102137 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:39:14.165642 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.165564 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/365e4722-bc8a-4e6a-b08d-25ffb2caede9-tls-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:39:14.165642 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.165590 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9n6\" (UniqueName: \"kubernetes.io/projected/365e4722-bc8a-4e6a-b08d-25ffb2caede9-kube-api-access-8v9n6\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:39:14.619256 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.619226 2569 generic.go:356] "Generic (PLEG): container finished" podID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" containerID="42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f" exitCode=0 Jul 07 16:39:14.619755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.619268 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" Jul 07 16:39:14.619755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.619319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" event={"ID":"365e4722-bc8a-4e6a-b08d-25ffb2caede9","Type":"ContainerDied","Data":"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f"} Jul 07 16:39:14.619755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.619363 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dc887ffcb-rtvb8" event={"ID":"365e4722-bc8a-4e6a-b08d-25ffb2caede9","Type":"ContainerDied","Data":"29def7103b3e1b38a6dcd6e517cba1da620e6b752aebea8e635d46f8f3418cc1"} Jul 07 16:39:14.619755 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.619385 2569 scope.go:117] "RemoveContainer" containerID="42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f" Jul 07 16:39:14.629740 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.629719 2569 scope.go:117] "RemoveContainer" containerID="42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f" Jul 07 16:39:14.630034 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:39:14.630016 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f\": container with ID starting with 42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f not found: ID does not exist" containerID="42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f" Jul 07 16:39:14.630108 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.630042 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f"} err="failed to get container status \"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f\": rpc error: code = NotFound desc = could not find container \"42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f\": container with ID starting with 42ec5bbeb7a519c2ac324e888609b0d3cf64e6d9f333ab095d9f67337722427f not found: ID does not exist" Jul 07 16:39:14.642241 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.642213 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:39:14.649715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:14.649691 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dc887ffcb-rtvb8"] Jul 07 16:39:16.268527 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:16.268499 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" path="/var/lib/kubelet/pods/365e4722-bc8a-4e6a-b08d-25ffb2caede9/volumes" Jul 07 16:39:22.225829 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:39:22.225786 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:40:25.284319 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.284276 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:40:25.375802 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.375764 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:40:25.376032 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.376008 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-56dc7d6c67-975nh" podUID="b72cca08-0381-40dc-a420-e5adeaa1a33e" containerName="manager" containerID="cri-o://0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4" gracePeriod=30 Jul 07 16:40:25.623478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.623454 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:40:25.655094 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.655073 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca\") pod \"b72cca08-0381-40dc-a420-e5adeaa1a33e\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " Jul 07 16:40:25.655209 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.655100 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert\") pod \"b72cca08-0381-40dc-a420-e5adeaa1a33e\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " Jul 07 16:40:25.655209 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.655128 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb7bj\" (UniqueName: \"kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj\") pod \"b72cca08-0381-40dc-a420-e5adeaa1a33e\" (UID: \"b72cca08-0381-40dc-a420-e5adeaa1a33e\") " Jul 07 16:40:25.655699 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.655413 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca" (OuterVolumeSpecName: "service-ca") pod "b72cca08-0381-40dc-a420-e5adeaa1a33e" (UID: "b72cca08-0381-40dc-a420-e5adeaa1a33e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:40:25.657590 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.657529 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b72cca08-0381-40dc-a420-e5adeaa1a33e" (UID: "b72cca08-0381-40dc-a420-e5adeaa1a33e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:40:25.658102 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.658065 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj" (OuterVolumeSpecName: "kube-api-access-pb7bj") pod "b72cca08-0381-40dc-a420-e5adeaa1a33e" (UID: "b72cca08-0381-40dc-a420-e5adeaa1a33e"). InnerVolumeSpecName "kube-api-access-pb7bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:40:25.756078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.756048 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b72cca08-0381-40dc-a420-e5adeaa1a33e-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:40:25.756078 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.756074 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b72cca08-0381-40dc-a420-e5adeaa1a33e-webhook-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:40:25.756202 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:25.756084 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7bj\" (UniqueName: \"kubernetes.io/projected/b72cca08-0381-40dc-a420-e5adeaa1a33e-kube-api-access-pb7bj\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:40:26.027872 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.027814 2569 generic.go:356] "Generic (PLEG): container finished" podID="b72cca08-0381-40dc-a420-e5adeaa1a33e" containerID="0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4" exitCode=0 Jul 07 16:40:26.027998 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.027885 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-975nh" Jul 07 16:40:26.027998 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.027906 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-975nh" event={"ID":"b72cca08-0381-40dc-a420-e5adeaa1a33e","Type":"ContainerDied","Data":"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4"} Jul 07 16:40:26.027998 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.027977 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-975nh" event={"ID":"b72cca08-0381-40dc-a420-e5adeaa1a33e","Type":"ContainerDied","Data":"8409b18e4ebbc2b14168a0483ad1d03fe020577538929af7483560e0d11de269"} Jul 07 16:40:26.028163 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.028009 2569 scope.go:117] "RemoveContainer" containerID="0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4" Jul 07 16:40:26.038102 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.038087 2569 scope.go:117] "RemoveContainer" containerID="0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4" Jul 07 16:40:26.038310 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:40:26.038294 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4\": container with ID starting with 0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4 not found: ID does not exist" containerID="0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4" Jul 07 16:40:26.038376 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.038325 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4"} err="failed to get container status \"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4\": rpc error: code = NotFound desc = could not find container \"0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4\": container with ID starting with 0acd43b5a1b77f10449a4e282b15e97848e19470906fff23070631739a6ac9f4 not found: ID does not exist" Jul 07 16:40:26.058464 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.058439 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:40:26.061240 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.061222 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-975nh"] Jul 07 16:40:26.268695 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.268666 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72cca08-0381-40dc-a420-e5adeaa1a33e" path="/var/lib/kubelet/pods/b72cca08-0381-40dc-a420-e5adeaa1a33e/volumes" Jul 07 16:40:26.747824 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.747797 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:40:26.874713 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.874685 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-kcgvj"] Jul 07 16:40:26.875163 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875145 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72cca08-0381-40dc-a420-e5adeaa1a33e" containerName="manager" Jul 07 16:40:26.875261 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875166 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72cca08-0381-40dc-a420-e5adeaa1a33e" containerName="manager" Jul 07 16:40:26.875261 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875182 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" containerName="authorino" Jul 07 16:40:26.875261 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875191 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" containerName="authorino" Jul 07 16:40:26.875413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875294 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="365e4722-bc8a-4e6a-b08d-25ffb2caede9" containerName="authorino" Jul 07 16:40:26.875413 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.875311 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72cca08-0381-40dc-a420-e5adeaa1a33e" containerName="manager" Jul 07 16:40:26.879849 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.879818 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:26.882215 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.882196 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 16:40:26.882423 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.882409 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 16:40:26.883379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.883363 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sbxp6\"" Jul 07 16:40:26.891734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.891709 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-kcgvj"] Jul 07 16:40:26.966172 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.966146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54193f19-612e-452b-8914-07912920c6b7-service-ca\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:26.966282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.966189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54193f19-612e-452b-8914-07912920c6b7-webhook-cert\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:26.966345 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:26.966274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkr5g\" (UniqueName: \"kubernetes.io/projected/54193f19-612e-452b-8914-07912920c6b7-kube-api-access-gkr5g\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.067060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.067002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54193f19-612e-452b-8914-07912920c6b7-webhook-cert\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.067060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.067048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkr5g\" (UniqueName: \"kubernetes.io/projected/54193f19-612e-452b-8914-07912920c6b7-kube-api-access-gkr5g\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.067195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.067118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54193f19-612e-452b-8914-07912920c6b7-service-ca\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.067769 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.067752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54193f19-612e-452b-8914-07912920c6b7-service-ca\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.069383 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.069365 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/54193f19-612e-452b-8914-07912920c6b7-webhook-cert\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.076152 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.076131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkr5g\" (UniqueName: \"kubernetes.io/projected/54193f19-612e-452b-8914-07912920c6b7-kube-api-access-gkr5g\") pod \"maas-controller-56dc7d6c67-kcgvj\" (UID: \"54193f19-612e-452b-8914-07912920c6b7\") " pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.191069 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.191044 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:27.338069 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:27.338043 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56dc7d6c67-kcgvj"] Jul 07 16:40:27.339074 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:40:27.339049 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54193f19_612e_452b_8914_07912920c6b7.slice/crio-61e92bda8c59b9c2980696a5a2e2c42502f443a45a745bc3f2c60b884fb6c2dd WatchSource:0}: Error finding container 61e92bda8c59b9c2980696a5a2e2c42502f443a45a745bc3f2c60b884fb6c2dd: Status 404 returned error can't find the container with id 61e92bda8c59b9c2980696a5a2e2c42502f443a45a745bc3f2c60b884fb6c2dd Jul 07 16:40:28.042440 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:28.042408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" event={"ID":"54193f19-612e-452b-8914-07912920c6b7","Type":"ContainerStarted","Data":"0be9ae09aec3ae88c402d26559312a559a23490f30873e000c5a9808cec0bed8"} Jul 07 16:40:28.042757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:28.042459 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" event={"ID":"54193f19-612e-452b-8914-07912920c6b7","Type":"ContainerStarted","Data":"61e92bda8c59b9c2980696a5a2e2c42502f443a45a745bc3f2c60b884fb6c2dd"} Jul 07 16:40:28.042757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:28.042496 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:40:28.065829 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:28.065784 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" podStartSLOduration=1.449588395 podStartE2EDuration="2.065769484s" totalImagesPullingTime="616.181089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:40:28.064047042 +0000 UTC m=+1032.372886321" watchObservedRunningTime="2026-07-07 16:40:28.065769484 +0000 UTC m=+1032.374608809" Jul 07 16:40:38.414122 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:38.414077 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:40:38.477968 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:38.477934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:40:39.054179 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:40:39.054154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56dc7d6c67-kcgvj" Jul 07 16:41:15.037852 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:15.037759 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:15.261034 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:15.260996 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:15.419567 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:15.419528 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:22.305023 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:22.304990 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:22.516952 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:22.516915 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:22.609820 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:22.609770 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:22.709286 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:22.709263 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:25.056191 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:25.056157 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:29.370025 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:29.369993 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:29.484752 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:29.484721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:43.370491 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:43.370456 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:46.088081 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:46.088053 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:41:58.033332 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:41:58.033298 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:43:16.200710 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:43:16.200679 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:43:16.205649 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:43:16.205627 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:45:00.162653 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.162623 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724045-b6hdd"] Jul 07 16:45:00.166057 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.166041 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:00.168681 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.168659 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\"" Jul 07 16:45:00.217783 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.217754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blv5r\" (UniqueName: \"kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r\") pod \"maas-api-key-cleanup-29724045-b6hdd\" (UID: \"229e8f96-74fc-439e-b82c-8d4c45079524\") " pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:00.228191 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.228169 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724045-b6hdd"] Jul 07 16:45:00.318410 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.318381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blv5r\" (UniqueName: \"kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r\") pod \"maas-api-key-cleanup-29724045-b6hdd\" (UID: \"229e8f96-74fc-439e-b82c-8d4c45079524\") " pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:00.328189 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.328163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blv5r\" (UniqueName: \"kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r\") pod \"maas-api-key-cleanup-29724045-b6hdd\" (UID: \"229e8f96-74fc-439e-b82c-8d4c45079524\") " pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:00.476230 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.476164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:00.619979 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.619947 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724045-b6hdd"] Jul 07 16:45:00.621020 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:45:00.620977 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229e8f96_74fc_439e_b82c_8d4c45079524.slice/crio-a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e WatchSource:0}: Error finding container a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e: Status 404 returned error can't find the container with id a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e Jul 07 16:45:00.622954 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:00.622938 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:45:01.578291 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:01.578221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" event={"ID":"229e8f96-74fc-439e-b82c-8d4c45079524","Type":"ContainerStarted","Data":"a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e"} Jul 07 16:45:04.600178 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:04.600145 2569 generic.go:356] "Generic (PLEG): container finished" podID="229e8f96-74fc-439e-b82c-8d4c45079524" containerID="ba5fe2331b5102aa0aa774dbeea892458c9639a32a4a1db8da6d964dc1660235" exitCode=0 Jul 07 16:45:04.600577 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:04.600182 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" event={"ID":"229e8f96-74fc-439e-b82c-8d4c45079524","Type":"ContainerDied","Data":"ba5fe2331b5102aa0aa774dbeea892458c9639a32a4a1db8da6d964dc1660235"} Jul 07 16:45:05.737977 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:05.737947 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:05.868220 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:05.868163 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blv5r\" (UniqueName: \"kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r\") pod \"229e8f96-74fc-439e-b82c-8d4c45079524\" (UID: \"229e8f96-74fc-439e-b82c-8d4c45079524\") " Jul 07 16:45:05.870148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:05.870122 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r" (OuterVolumeSpecName: "kube-api-access-blv5r") pod "229e8f96-74fc-439e-b82c-8d4c45079524" (UID: "229e8f96-74fc-439e-b82c-8d4c45079524"). InnerVolumeSpecName "kube-api-access-blv5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:45:05.969017 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:05.968987 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blv5r\" (UniqueName: \"kubernetes.io/projected/229e8f96-74fc-439e-b82c-8d4c45079524-kube-api-access-blv5r\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:45:06.611571 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:06.611536 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" Jul 07 16:45:06.611571 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:06.611550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724045-b6hdd" event={"ID":"229e8f96-74fc-439e-b82c-8d4c45079524","Type":"ContainerDied","Data":"a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e"} Jul 07 16:45:06.611738 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:06.611607 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a83064453b0716bb77f3156dc224548c8165247b86c13826ee94ffef66f77e" Jul 07 16:45:11.447684 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:11.447646 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:45:27.430695 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:27.430654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:45:27.527517 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:45:27.527469 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:48:16.240949 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:48:16.240918 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:48:16.247867 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:48:16.247829 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:48:55.119174 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:48:55.119093 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:50:05.034934 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.034902 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c58cdc598-2pdkw"] Jul 07 16:50:05.037278 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.035458 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229e8f96-74fc-439e-b82c-8d4c45079524" containerName="cleanup" Jul 07 16:50:05.037278 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.035475 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="229e8f96-74fc-439e-b82c-8d4c45079524" containerName="cleanup" Jul 07 16:50:05.037278 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.035556 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="229e8f96-74fc-439e-b82c-8d4c45079524" containerName="cleanup" Jul 07 16:50:05.038416 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.038397 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.075057 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.075029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c58cdc598-2pdkw"] Jul 07 16:50:05.115700 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.115665 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:50:05.135871 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.135849 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-service-ca\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.135972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.135891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzt4\" (UniqueName: \"kubernetes.io/projected/724689cd-6362-4993-b489-5d07b366a4f4-kube-api-access-8qzt4\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.136024 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.135974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.136024 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.136003 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-oauth-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.136105 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.136039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-trusted-ca-bundle\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.136105 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.136071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-console-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.136105 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.136098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-oauth-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.190608 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.190586 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:50:05.236703 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-service-ca\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.236794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzt4\" (UniqueName: \"kubernetes.io/projected/724689cd-6362-4993-b489-5d07b366a4f4-kube-api-access-8qzt4\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.236794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.236794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236789 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-oauth-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.236935 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-trusted-ca-bundle\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.236935 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-console-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.236971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-oauth-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.237850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-console-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.238015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-service-ca\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.238141 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-trusted-ca-bundle\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.238339 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/724689cd-6362-4993-b489-5d07b366a4f4-oauth-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.240313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-serving-cert\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.246599 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.246581 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/724689cd-6362-4993-b489-5d07b366a4f4-console-oauth-config\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.250775 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.250750 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzt4\" (UniqueName: \"kubernetes.io/projected/724689cd-6362-4993-b489-5d07b366a4f4-kube-api-access-8qzt4\") pod \"console-6c58cdc598-2pdkw\" (UID: \"724689cd-6362-4993-b489-5d07b366a4f4\") " pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.348226 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.348205 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:05.501378 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:05.501353 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c58cdc598-2pdkw"] Jul 07 16:50:05.502958 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:50:05.502928 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724689cd_6362_4993_b489_5d07b366a4f4.slice/crio-ba480c3d29ea9ddb5d34e04d73c71366d28fdb660e61bec3ee35e91a8ebcc921 WatchSource:0}: Error finding container ba480c3d29ea9ddb5d34e04d73c71366d28fdb660e61bec3ee35e91a8ebcc921: Status 404 returned error can't find the container with id ba480c3d29ea9ddb5d34e04d73c71366d28fdb660e61bec3ee35e91a8ebcc921 Jul 07 16:50:06.322525 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:06.322484 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cdc598-2pdkw" event={"ID":"724689cd-6362-4993-b489-5d07b366a4f4","Type":"ContainerStarted","Data":"5fa6cf83fbdfe7f7fdafbc574d7dbb49759c5de9c3c7a5d9ecf3214e087992af"} Jul 07 16:50:06.322992 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:06.322534 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c58cdc598-2pdkw" event={"ID":"724689cd-6362-4993-b489-5d07b366a4f4","Type":"ContainerStarted","Data":"ba480c3d29ea9ddb5d34e04d73c71366d28fdb660e61bec3ee35e91a8ebcc921"} Jul 07 16:50:06.346706 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:06.346660 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c58cdc598-2pdkw" podStartSLOduration=1.3466408429999999 podStartE2EDuration="1.346640843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:50:06.344448481 +0000 UTC m=+1610.653287759" watchObservedRunningTime="2026-07-07 16:50:06.346640843 +0000 UTC m=+1610.655480121" Jul 07 16:50:15.348717 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.348679 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:15.349148 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.348737 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:15.353437 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.353414 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:15.373713 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.373693 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c58cdc598-2pdkw" Jul 07 16:50:15.439087 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.439055 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:50:15.687433 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:15.687345 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:50:17.353149 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:17.353106 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:50:40.459462 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.459400 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d5ffdd8fd-vwhht" podUID="80e23d1b-7ccb-49e6-976f-668876876226" containerName="console" containerID="cri-o://c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d" gracePeriod=15 Jul 07 16:50:40.694958 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.694933 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:50:40.713689 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713631 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.713689 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713678 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.713903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713698 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spnr8\" (UniqueName: \"kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.713903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713717 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.713903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713862 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.714075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713906 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.714075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.713944 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca\") pod \"80e23d1b-7ccb-49e6-976f-668876876226\" (UID: \"80e23d1b-7ccb-49e6-976f-668876876226\") " Jul 07 16:50:40.714075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714055 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:50:40.714243 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714096 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config" (OuterVolumeSpecName: "console-config") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:50:40.714243 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714230 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:50:40.714449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714422 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-trusted-ca-bundle\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.714519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714447 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-console-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.714519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714464 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-oauth-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.714519 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.714445 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca" (OuterVolumeSpecName: "service-ca") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:50:40.715992 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.715966 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:50:40.716084 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.715996 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:50:40.716084 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.716035 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8" (OuterVolumeSpecName: "kube-api-access-spnr8") pod "80e23d1b-7ccb-49e6-976f-668876876226" (UID: "80e23d1b-7ccb-49e6-976f-668876876226"). InnerVolumeSpecName "kube-api-access-spnr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:50:40.814828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.814801 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spnr8\" (UniqueName: \"kubernetes.io/projected/80e23d1b-7ccb-49e6-976f-668876876226-kube-api-access-spnr8\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.814942 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.814826 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-oauth-config\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.814942 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.814864 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80e23d1b-7ccb-49e6-976f-668876876226-console-serving-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:40.814942 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:40.814879 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80e23d1b-7ccb-49e6-976f-668876876226-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:50:41.525972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.525942 2569 generic.go:356] "Generic (PLEG): container finished" podID="80e23d1b-7ccb-49e6-976f-668876876226" containerID="c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d" exitCode=0 Jul 07 16:50:41.526341 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.525998 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5ffdd8fd-vwhht" Jul 07 16:50:41.526341 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.526032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5ffdd8fd-vwhht" event={"ID":"80e23d1b-7ccb-49e6-976f-668876876226","Type":"ContainerDied","Data":"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d"} Jul 07 16:50:41.526341 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.526077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5ffdd8fd-vwhht" event={"ID":"80e23d1b-7ccb-49e6-976f-668876876226","Type":"ContainerDied","Data":"e0a73b9ee0c2927bbdc0428541f7c650007f9a3347c8c34653e8f92a9187b2a5"} Jul 07 16:50:41.526341 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.526097 2569 scope.go:117] "RemoveContainer" containerID="c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d" Jul 07 16:50:41.535252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.535237 2569 scope.go:117] "RemoveContainer" containerID="c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d" Jul 07 16:50:41.535501 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:50:41.535483 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d\": container with ID starting with c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d not found: ID does not exist" containerID="c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d" Jul 07 16:50:41.535539 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.535510 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d"} err="failed to get container status \"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d\": rpc error: code = NotFound desc = could not find container \"c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d\": container with ID starting with c5c04a696b54481e67c7f105602b66b487da90e89e856f74ec3e552e0903021d not found: ID does not exist" Jul 07 16:50:41.562328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.562304 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:50:41.570813 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:41.570793 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5ffdd8fd-vwhht"] Jul 07 16:50:42.268091 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:50:42.268061 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e23d1b-7ccb-49e6-976f-668876876226" path="/var/lib/kubelet/pods/80e23d1b-7ccb-49e6-976f-668876876226/volumes" Jul 07 16:52:46.185508 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:52:46.185467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:52:56.023706 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:52:56.023671 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:04.319866 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:04.319811 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:09.504903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:09.504806 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:11.343516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:11.343469 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:12.605621 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:12.605572 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:12.690205 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:12.690126 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:13.181918 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:13.181886 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:16.282282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:16.282251 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:53:16.291028 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:16.291001 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:53:19.603000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:19.602962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:19.772824 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:19.772791 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:19.948734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:19.948704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:20.057037 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:20.057000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:21.163809 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.163781 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:21.334115 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.334080 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:21.334557 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.334538 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e23d1b-7ccb-49e6-976f-668876876226" containerName="console" Jul 07 16:53:21.334652 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.334560 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e23d1b-7ccb-49e6-976f-668876876226" containerName="console" Jul 07 16:53:21.334715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.334659 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e23d1b-7ccb-49e6-976f-668876876226" containerName="console" Jul 07 16:53:21.337702 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.337680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.341051 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.341031 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\"" Jul 07 16:53:21.341170 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.341063 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-6306fce7-serving-cert\"" Jul 07 16:53:21.364719 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.364694 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:21.410142 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.410119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.410252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.410170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28xk\" (UniqueName: \"kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.410252 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.410190 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.481506 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.481445 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:21.510622 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.510603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.510734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.510651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28xk\" (UniqueName: \"kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.510734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.510672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.511222 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.511201 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.513131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.513113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.537380 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.537356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28xk\" (UniqueName: \"kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk\") pod \"maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.630486 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.630463 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:21.649537 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.649511 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:21.793916 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.793889 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:21.795616 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:53:21.795579 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18587359_326f_4827_b1f6_b07e4e960eea.slice/crio-d77e10a63caf585d3556c9bbbc766b06fed1fe3df5821db1d4fdfed7a81e19d1 WatchSource:0}: Error finding container d77e10a63caf585d3556c9bbbc766b06fed1fe3df5821db1d4fdfed7a81e19d1: Status 404 returned error can't find the container with id d77e10a63caf585d3556c9bbbc766b06fed1fe3df5821db1d4fdfed7a81e19d1 Jul 07 16:53:21.797379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:21.797361 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:53:22.205946 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:22.205919 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:22.302757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:22.302726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:22.438020 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:22.437989 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" event={"ID":"18587359-326f-4827-b1f6-b07e4e960eea","Type":"ContainerStarted","Data":"d77e10a63caf585d3556c9bbbc766b06fed1fe3df5821db1d4fdfed7a81e19d1"} Jul 07 16:53:24.030789 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:24.030756 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:24.451857 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:24.451802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" event={"ID":"18587359-326f-4827-b1f6-b07e4e960eea","Type":"ContainerStarted","Data":"195b93ccaf551d87a94fccadb825e258be4c2b88c2cf587adb2bfcb535bf4dfe"} Jul 07 16:53:24.452019 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:24.451966 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:24.485973 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:24.485926 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" podStartSLOduration=1.813181213 podStartE2EDuration="3.485912533s" totalImagesPullingTime="1.67273132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:24.481496798 +0000 UTC m=+1808.790336075" watchObservedRunningTime="2026-07-07 16:53:24.485912533 +0000 UTC m=+1808.794751811" Jul 07 16:53:29.267220 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.267187 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:29.267677 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.267366 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" podUID="18587359-326f-4827-b1f6-b07e4e960eea" containerName="maas-api" containerID="cri-o://195b93ccaf551d87a94fccadb825e258be4c2b88c2cf587adb2bfcb535bf4dfe" gracePeriod=30 Jul 07 16:53:29.272960 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.272934 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:29.305927 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.305904 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:29.375733 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.375711 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:29.482187 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.482161 2569 generic.go:356] "Generic (PLEG): container finished" podID="18587359-326f-4827-b1f6-b07e4e960eea" containerID="195b93ccaf551d87a94fccadb825e258be4c2b88c2cf587adb2bfcb535bf4dfe" exitCode=0 Jul 07 16:53:29.482317 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.482195 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" event={"ID":"18587359-326f-4827-b1f6-b07e4e960eea","Type":"ContainerDied","Data":"195b93ccaf551d87a94fccadb825e258be4c2b88c2cf587adb2bfcb535bf4dfe"} Jul 07 16:53:29.518688 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.518639 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:29.540021 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.539999 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:29.622207 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.622176 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:29.675318 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.675288 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls\") pod \"18587359-326f-4827-b1f6-b07e4e960eea\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " Jul 07 16:53:29.675449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.675377 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca\") pod \"18587359-326f-4827-b1f6-b07e4e960eea\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " Jul 07 16:53:29.675515 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.675451 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h28xk\" (UniqueName: \"kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk\") pod \"18587359-326f-4827-b1f6-b07e4e960eea\" (UID: \"18587359-326f-4827-b1f6-b07e4e960eea\") " Jul 07 16:53:29.675746 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.675724 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18587359-326f-4827-b1f6-b07e4e960eea" (UID: "18587359-326f-4827-b1f6-b07e4e960eea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:53:29.677460 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.677440 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk" (OuterVolumeSpecName: "kube-api-access-h28xk") pod "18587359-326f-4827-b1f6-b07e4e960eea" (UID: "18587359-326f-4827-b1f6-b07e4e960eea"). InnerVolumeSpecName "kube-api-access-h28xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:53:29.677556 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.677487 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18587359-326f-4827-b1f6-b07e4e960eea" (UID: "18587359-326f-4827-b1f6-b07e4e960eea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:53:29.776786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.776726 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18587359-326f-4827-b1f6-b07e4e960eea-openshift-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:53:29.776786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.776749 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28xk\" (UniqueName: \"kubernetes.io/projected/18587359-326f-4827-b1f6-b07e4e960eea-kube-api-access-h28xk\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:53:29.776786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:29.776759 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18587359-326f-4827-b1f6-b07e4e960eea-maas-api-tls\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:53:30.488645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:30.488618 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" event={"ID":"18587359-326f-4827-b1f6-b07e4e960eea","Type":"ContainerDied","Data":"d77e10a63caf585d3556c9bbbc766b06fed1fe3df5821db1d4fdfed7a81e19d1"} Jul 07 16:53:30.489092 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:30.488675 2569 scope.go:117] "RemoveContainer" containerID="195b93ccaf551d87a94fccadb825e258be4c2b88c2cf587adb2bfcb535bf4dfe" Jul 07 16:53:30.489092 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:30.488628 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c" Jul 07 16:53:30.518524 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:30.518501 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:30.524763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:30.524735 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-6306fce7-69fdf8d787-n7h4c"] Jul 07 16:53:31.362641 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:31.362609 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:32.267881 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:32.267827 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18587359-326f-4827-b1f6-b07e4e960eea" path="/var/lib/kubelet/pods/18587359-326f-4827-b1f6-b07e4e960eea/volumes" Jul 07 16:53:36.270448 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:36.270385 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:36.368274 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:36.368239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:36.517706 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:36.517674 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:36.693626 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:36.693598 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:37.744933 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:37.744903 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:37.870956 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:37.870928 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:37.950656 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:37.950629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:38.180754 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:38.180727 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:38.536698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:38.536632 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:38.639701 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:38.639676 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:40.183060 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:40.183025 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:45.962698 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:45.962668 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:46.060991 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:46.060962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:47.230306 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:47.230277 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:47.338327 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:47.338300 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:51.507786 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:51.507760 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:51.893014 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:51.892986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:51.956963 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:51.956935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:53.318578 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.318541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:53.401998 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.401969 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:53.547688 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.547658 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:53:53.548156 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.548139 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18587359-326f-4827-b1f6-b07e4e960eea" containerName="maas-api" Jul 07 16:53:53.548260 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.548158 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="18587359-326f-4827-b1f6-b07e4e960eea" containerName="maas-api" Jul 07 16:53:53.548317 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.548280 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="18587359-326f-4827-b1f6-b07e4e960eea" containerName="maas-api" Jul 07 16:53:53.551959 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.551938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.555183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.555163 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f2f03512-openshift-default-dockercfg-vhvws\"" Jul 07 16:53:53.555183 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.555170 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 16:53:53.585876 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.585827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:53:53.611047 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.611018 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:53.665120 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665217 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665217 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665217 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665160 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xhd9\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665322 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665391 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665428 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.665428 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.665423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.706645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.706603 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:53.766328 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xhd9\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766416 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766416 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766505 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766505 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766685 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766685 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766685 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766600 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766685 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766635 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766954 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.766954 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766884 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.767071 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.766999 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.767127 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.767082 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.767296 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.767277 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.768709 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.768687 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.769075 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.769055 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.776238 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.776210 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xhd9\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.776436 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.776418 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token\") pod \"e2e-derive-f2f03512-openshift-default-55998f4659-tkthx\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:53.862591 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:53.862536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:54.009844 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:54.009806 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:53:54.011135 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:53:54.011112 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14609c64_f394_4083_a136_58affc089379.slice/crio-c5a93bb036d6f154c87aac7d1d95ea9e6701e56868abb314009451a1cb963e17 WatchSource:0}: Error finding container c5a93bb036d6f154c87aac7d1d95ea9e6701e56868abb314009451a1cb963e17: Status 404 returned error can't find the container with id c5a93bb036d6f154c87aac7d1d95ea9e6701e56868abb314009451a1cb963e17 Jul 07 16:53:54.623904 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:54.623829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" event={"ID":"14609c64-f394-4083-a136-58affc089379","Type":"ContainerStarted","Data":"c5a93bb036d6f154c87aac7d1d95ea9e6701e56868abb314009451a1cb963e17"} Jul 07 16:53:54.713206 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:54.713173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:54.812766 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:54.812732 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:54.962254 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:54.962142 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:55.419999 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:55.419953 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:55.564565 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:55.564531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:55.663410 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:55.663381 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:56.608071 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:56.608031 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:53:56.608207 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:56.608120 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:53:56.608207 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:56.608165 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:53:57.124512 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:57.124486 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:53:57.644176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:57.644144 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" event={"ID":"14609c64-f394-4083-a136-58affc089379","Type":"ContainerStarted","Data":"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5"} Jul 07 16:53:57.674145 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:57.674092 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" podStartSLOduration=2.079859903 podStartE2EDuration="4.674072366s" totalImagesPullingTime="2.594212463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:53:57.672610083 +0000 UTC m=+1841.981449363" watchObservedRunningTime="2026-07-07 16:53:57.674072366 +0000 UTC m=+1841.982911643" Jul 07 16:53:57.862905 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:57.862879 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:57.867801 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:57.867779 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:58.650251 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:58.650214 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:53:58.651119 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:53:58.651099 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:54:02.383167 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.383129 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:02.458972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.458938 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:54:02.459155 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.459131 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" podUID="14609c64-f394-4083-a136-58affc089379" containerName="istio-proxy" containerID="cri-o://2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5" gracePeriod=30 Jul 07 16:54:02.478171 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.478148 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:02.583037 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.582988 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:02.687915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:02.685080 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:05.978890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:05.978816 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:08.763526 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:08.763490 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:08.864787 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:08.864736 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:09.314161 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.314133 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:54:09.318036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.317976 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:09.407725 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.407879 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407752 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.407879 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407778 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.407879 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.407879 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407818 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.407879 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407865 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.408164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407914 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.408164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407940 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.408164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.407973 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xhd9\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9\") pod \"14609c64-f394-4083-a136-58affc089379\" (UID: \"14609c64-f394-4083-a136-58affc089379\") " Jul 07 16:54:09.408164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408079 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:54:09.408371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408231 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:54:09.408371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408330 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:54:09.408473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408363 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-credential-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.408473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408388 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-certs\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.408573 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408526 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:54:09.408629 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.408574 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data" (OuterVolumeSpecName: "istio-data") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:54:09.410300 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.410273 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:54:09.410300 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.410288 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9" (OuterVolumeSpecName: "kube-api-access-4xhd9") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "kube-api-access-4xhd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:54:09.410449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.410303 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token" (OuterVolumeSpecName: "istio-token") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:54:09.410449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.410320 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14609c64-f394-4083-a136-58affc089379" (UID: "14609c64-f394-4083-a136-58affc089379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:54:09.418715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.418690 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:09.509106 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509086 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-istio-token\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509106 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509104 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xhd9\" (UniqueName: \"kubernetes.io/projected/14609c64-f394-4083-a136-58affc089379-kube-api-access-4xhd9\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509114 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-data\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509124 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14609c64-f394-4083-a136-58affc089379-istio-podinfo\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509132 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-workload-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509140 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14609c64-f394-4083-a136-58affc089379-istio-envoy\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.509234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.509147 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14609c64-f394-4083-a136-58affc089379-istiod-ca-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:54:09.513757 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.513737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:09.731663 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.731634 2569 generic.go:356] "Generic (PLEG): container finished" podID="14609c64-f394-4083-a136-58affc089379" containerID="2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5" exitCode=0 Jul 07 16:54:09.731794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.731693 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" Jul 07 16:54:09.731794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.731690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" event={"ID":"14609c64-f394-4083-a136-58affc089379","Type":"ContainerDied","Data":"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5"} Jul 07 16:54:09.731794 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.731776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx" event={"ID":"14609c64-f394-4083-a136-58affc089379","Type":"ContainerDied","Data":"c5a93bb036d6f154c87aac7d1d95ea9e6701e56868abb314009451a1cb963e17"} Jul 07 16:54:09.731966 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.731808 2569 scope.go:117] "RemoveContainer" containerID="2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5" Jul 07 16:54:09.741763 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.741747 2569 scope.go:117] "RemoveContainer" containerID="2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5" Jul 07 16:54:09.742027 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:54:09.742007 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5\": container with ID starting with 2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5 not found: ID does not exist" containerID="2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5" Jul 07 16:54:09.742097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.742036 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5"} err="failed to get container status \"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5\": rpc error: code = NotFound desc = could not find container \"2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5\": container with ID starting with 2929cbc7b732f4d1d12b5a54cea19b4a50e1fd8a7f19144c61f17085c91fb7b5 not found: ID does not exist" Jul 07 16:54:09.758379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.758327 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:54:09.764861 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:09.764817 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f2f03512-openshift-default-55998f4659-tkthx"] Jul 07 16:54:10.267918 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:10.267882 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14609c64-f394-4083-a136-58affc089379" path="/var/lib/kubelet/pods/14609c64-f394-4083-a136-58affc089379/volumes" Jul 07 16:54:11.178171 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:11.178127 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:17.664707 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:17.664667 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:17.841346 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:17.841317 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:18.106213 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:18.106169 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:18.559738 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:18.559665 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:44.652284 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:44.652171 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:50.536661 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:50.536629 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:50.638900 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:50.638870 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:50.736958 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:50.736933 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:54:50.837260 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:54:50.837229 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:06.873043 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:06.873005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:12.979890 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:12.979852 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:13.077596 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:13.077552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:13.179594 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:13.179572 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:13.288038 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:13.287959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:23.827931 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:23.827888 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:25.296548 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:25.296510 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:30.953348 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:30.953313 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:31.063396 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:31.063365 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:31.191172 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:31.191131 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:31.292915 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:31.292853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:32.212335 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:32.212306 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:32.309485 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:32.309455 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:32.414882 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:32.414823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:34.293513 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:34.293480 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:40.309059 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:40.309025 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:40.371318 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:40.371285 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:41.031645 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:41.031614 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:47.182510 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:47.182476 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:47.283769 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:47.283741 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:47.540861 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:47.540775 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:47.850036 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:47.849985 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:48.000188 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:48.000159 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:49.000979 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:49.000948 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:49.099877 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:49.099819 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.041402 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.041360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.114420 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.114393 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:55:51.114856 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.114829 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14609c64-f394-4083-a136-58affc089379" containerName="istio-proxy" Jul 07 16:55:51.114911 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.114860 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="14609c64-f394-4083-a136-58affc089379" containerName="istio-proxy" Jul 07 16:55:51.114969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.114957 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="14609c64-f394-4083-a136-58affc089379" containerName="istio-proxy" Jul 07 16:55:51.118160 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.118136 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.120595 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.120569 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-79f2ee03-serving-cert\"" Jul 07 16:55:51.120741 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.120648 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\"" Jul 07 16:55:51.130121 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.130100 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:55:51.134459 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.134439 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.195449 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.195423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z224n\" (UniqueName: \"kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.195594 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.195473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.195594 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.195529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.238530 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.238502 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.296726 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.296656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.296726 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.296703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.296943 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.296749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z224n\" (UniqueName: \"kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.297352 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.297330 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.299129 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.299109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.307196 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.307172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z224n\" (UniqueName: \"kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n\") pod \"maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.336349 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.336325 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.431618 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.431598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:51.590443 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:55:51.590417 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6534a0c9_68aa_4980_a7c1_4ea246103d93.slice/crio-0f2b6c8a605fb3c1bb36b0e76045c3dfe1d4bb8eab0376e461fa6d783b33ecfc WatchSource:0}: Error finding container 0f2b6c8a605fb3c1bb36b0e76045c3dfe1d4bb8eab0376e461fa6d783b33ecfc: Status 404 returned error can't find the container with id 0f2b6c8a605fb3c1bb36b0e76045c3dfe1d4bb8eab0376e461fa6d783b33ecfc Jul 07 16:55:51.598950 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.598925 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:55:51.761615 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.761592 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:51.857662 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:51.857601 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:52.293636 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:52.293604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" event={"ID":"6534a0c9-68aa-4980-a7c1-4ea246103d93","Type":"ContainerStarted","Data":"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd"} Jul 07 16:55:52.294119 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:52.293651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" event={"ID":"6534a0c9-68aa-4980-a7c1-4ea246103d93","Type":"ContainerStarted","Data":"0f2b6c8a605fb3c1bb36b0e76045c3dfe1d4bb8eab0376e461fa6d783b33ecfc"} Jul 07 16:55:52.294119 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:52.293775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:55:52.327982 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:52.327936 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" podStartSLOduration=0.81648795 podStartE2EDuration="1.327917424s" totalImagesPullingTime="511.429474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:55:52.327342795 +0000 UTC m=+1956.636182072" watchObservedRunningTime="2026-07-07 16:55:52.327917424 +0000 UTC m=+1956.636756703" Jul 07 16:55:55.511666 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:55.511628 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:56.276370 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:56.276338 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:56.378316 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:56.378288 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:56.799945 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:56.799918 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:55:58.306166 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:55:58.306135 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:56:03.948812 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:03.948751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:04.092529 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:04.092496 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:04.494184 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:04.494155 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:04.931173 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:04.931140 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:04.959898 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:04.959864 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:56:04.960287 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:04.960044 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" podUID="6534a0c9-68aa-4980-a7c1-4ea246103d93" containerName="maas-api" containerID="cri-o://49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd" gracePeriod=30 Jul 07 16:56:05.047170 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.047138 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:05.205233 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.205210 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:56:05.305544 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.305515 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca\") pod \"6534a0c9-68aa-4980-a7c1-4ea246103d93\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " Jul 07 16:56:05.305691 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.305566 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z224n\" (UniqueName: \"kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n\") pod \"6534a0c9-68aa-4980-a7c1-4ea246103d93\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " Jul 07 16:56:05.305691 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.305595 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls\") pod \"6534a0c9-68aa-4980-a7c1-4ea246103d93\" (UID: \"6534a0c9-68aa-4980-a7c1-4ea246103d93\") " Jul 07 16:56:05.305940 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.305914 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6534a0c9-68aa-4980-a7c1-4ea246103d93" (UID: "6534a0c9-68aa-4980-a7c1-4ea246103d93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:56:05.307663 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.307638 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6534a0c9-68aa-4980-a7c1-4ea246103d93" (UID: "6534a0c9-68aa-4980-a7c1-4ea246103d93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:56:05.307759 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.307679 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n" (OuterVolumeSpecName: "kube-api-access-z224n") pod "6534a0c9-68aa-4980-a7c1-4ea246103d93" (UID: "6534a0c9-68aa-4980-a7c1-4ea246103d93"). InnerVolumeSpecName "kube-api-access-z224n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:56:05.368652 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.368623 2569 generic.go:356] "Generic (PLEG): container finished" podID="6534a0c9-68aa-4980-a7c1-4ea246103d93" containerID="49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd" exitCode=0 Jul 07 16:56:05.368744 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.368684 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" Jul 07 16:56:05.368744 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.368713 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" event={"ID":"6534a0c9-68aa-4980-a7c1-4ea246103d93","Type":"ContainerDied","Data":"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd"} Jul 07 16:56:05.368828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.368770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf" event={"ID":"6534a0c9-68aa-4980-a7c1-4ea246103d93","Type":"ContainerDied","Data":"0f2b6c8a605fb3c1bb36b0e76045c3dfe1d4bb8eab0376e461fa6d783b33ecfc"} Jul 07 16:56:05.368828 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.368796 2569 scope.go:117] "RemoveContainer" containerID="49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd" Jul 07 16:56:05.378304 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.378287 2569 scope.go:117] "RemoveContainer" containerID="49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd" Jul 07 16:56:05.378554 ip-10-0-142-74 kubenswrapper[2569]: E0707 16:56:05.378538 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd\": container with ID starting with 49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd not found: ID does not exist" containerID="49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd" Jul 07 16:56:05.378630 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.378571 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd"} err="failed to get container status \"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd\": rpc error: code = NotFound desc = could not find container \"49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd\": container with ID starting with 49c23e74d29a160a3ddf032509b2be27d2a82b257b863979fe6bbbd7d246c7bd not found: ID does not exist" Jul 07 16:56:05.407017 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.406994 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6534a0c9-68aa-4980-a7c1-4ea246103d93-openshift-service-ca\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:56:05.407017 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.407015 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z224n\" (UniqueName: \"kubernetes.io/projected/6534a0c9-68aa-4980-a7c1-4ea246103d93-kube-api-access-z224n\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:56:05.407176 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.407025 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6534a0c9-68aa-4980-a7c1-4ea246103d93-maas-api-tls\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 16:56:05.408379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.408357 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:56:05.431388 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:05.431367 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-79f2ee03-5b5984bd4c-5ghpf"] Jul 07 16:56:06.268990 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:06.268952 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6534a0c9-68aa-4980-a7c1-4ea246103d93" path="/var/lib/kubelet/pods/6534a0c9-68aa-4980-a7c1-4ea246103d93/volumes" Jul 07 16:56:11.510381 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:11.510304 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:11.611456 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:11.611426 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:11.783623 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:11.783539 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:12.050299 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:12.050223 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:12.230994 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:12.230962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:12.349582 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:12.349552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:12.444972 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:12.444930 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:18.932593 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:18.932552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:19.098679 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:19.098648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:19.226344 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:19.226258 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:19.464610 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:19.464570 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:25.815379 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:25.815348 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:25.917687 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:25.917650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:32.949934 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:32.949898 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:33.382097 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:33.382058 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:33.612847 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:33.612795 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:36.896777 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:36.896745 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:37.037000 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:37.036967 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:37.264402 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:37.264327 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:37.455161 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:37.455137 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:38.201565 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:38.201535 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:38.364624 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:38.364597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:41.407787 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:41.407750 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:56:47.716721 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:56:47.716692 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:08.865805 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:08.865769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:09.269193 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:09.269117 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:14.981715 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:14.981680 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:20.035305 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:20.035271 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:24.503112 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:24.503079 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:56.422131 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:56.422048 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:57:58.474964 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:57:58.474933 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:04.609996 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:04.609959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:04.793035 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:04.792997 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:05.028797 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:05.028708 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:05.248904 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:05.248863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:06.364586 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:06.364554 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:06.470969 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:06.470936 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:06.568258 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:06.568233 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:07.323448 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:07.323398 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:07.424184 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:07.424149 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:07.534110 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:07.534082 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:09.094392 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:09.094355 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:16.322817 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:16.322789 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:58:16.333013 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:16.332990 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 16:58:16.838699 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:16.838666 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:17.032197 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:17.032172 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:17.494903 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:17.494874 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:23.797516 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:23.797479 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:23.910511 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:23.910486 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:24.009307 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:24.009282 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:34.592635 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:34.592599 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:34.952113 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:34.952035 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:58:55.980691 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:58:55.980645 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:02.726203 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:02.726160 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:02.837970 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:02.837935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:03.124195 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:03.124161 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:03.519679 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:03.519610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:03.919815 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:03.919787 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:06.672378 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:06.672340 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:06.858180 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:06.858135 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:06.990941 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:06.990868 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:07.422342 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:07.422297 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:07.534929 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:07.534903 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:10.471941 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:10.471903 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:11.194276 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.194244 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:11.534108 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.534030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:11.890366 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.890333 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 16:59:11.890737 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.890723 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6534a0c9-68aa-4980-a7c1-4ea246103d93" containerName="maas-api" Jul 07 16:59:11.890737 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.890738 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6534a0c9-68aa-4980-a7c1-4ea246103d93" containerName="maas-api" Jul 07 16:59:11.890883 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.890802 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6534a0c9-68aa-4980-a7c1-4ea246103d93" containerName="maas-api" Jul 07 16:59:11.893443 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.893427 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.898425 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.898410 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4b5d6a-openshift-default-dockercfg-cbqkh\"" Jul 07 16:59:11.899070 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.899052 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 16:59:11.958855 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.958823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:11.990164 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990175 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990282 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990409 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990406 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990436 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:11.990473 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.990468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxrc\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.002308 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:11.995738 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 16:59:12.090847 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.090816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.090941 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.090866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.090941 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.090915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091047 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.090945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091047 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.090983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091047 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091204 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091263 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091263 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091253 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxrc\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091371 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091601 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091708 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091598 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091708 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091684 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.091800 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.091722 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.093419 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.093390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.093526 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.093507 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.116175 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.116152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxrc\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.117057 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.117037 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token\") pod \"e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.203990 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.203931 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:12.345619 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.345596 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 16:59:12.347117 ip-10-0-142-74 kubenswrapper[2569]: W0707 16:59:12.347088 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246866d5_c9f3_41a7_83c6_86de36010631.slice/crio-66d11c426fbc996b0e9e5167266d5646da70e87847eb4e632d34875603a422f4 WatchSource:0}: Error finding container 66d11c426fbc996b0e9e5167266d5646da70e87847eb4e632d34875603a422f4: Status 404 returned error can't find the container with id 66d11c426fbc996b0e9e5167266d5646da70e87847eb4e632d34875603a422f4 Jul 07 16:59:12.349234 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.349199 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:12.349337 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.349266 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:12.349337 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.349295 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:12.422331 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.422304 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerStarted","Data":"66d11c426fbc996b0e9e5167266d5646da70e87847eb4e632d34875603a422f4"} Jul 07 16:59:12.813850 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:12.813805 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:13.430430 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:13.430398 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerStarted","Data":"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00"} Jul 07 16:59:13.464478 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:13.464430 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podStartSLOduration=2.464414604 podStartE2EDuration="2.464414604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:59:13.463310947 +0000 UTC m=+2157.772150227" watchObservedRunningTime="2026-07-07 16:59:13.464414604 +0000 UTC m=+2157.773253881" Jul 07 16:59:14.204356 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:14.204321 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:14.209427 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:14.209403 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:14.435612 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:14.435588 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:14.436323 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:14.436301 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:15.549811 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:15.549777 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:15.687553 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:15.687524 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:15.838347 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:15.838306 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:16.192061 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:16.191964 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:16.424032 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:16.424005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:16.527208 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:16.527102 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:19.991247 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:19.991211 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:24.206634 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:24.206588 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:29.683593 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:29.683541 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:29.784704 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:29.784675 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:30.476976 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:30.476938 2569 patch_prober.go:28] interesting pod/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 16:59:30.477151 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:30.477011 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 16:59:31.537284 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.537256 2569 generic.go:356] "Generic (PLEG): container finished" podID="246866d5-c9f3-41a7-83c6-86de36010631" containerID="03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00" exitCode=0 Jul 07 16:59:31.537675 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.537332 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerDied","Data":"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00"} Jul 07 16:59:31.537675 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.537670 2569 scope.go:117] "RemoveContainer" containerID="03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00" Jul 07 16:59:31.538672 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.538647 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:31.538795 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.538707 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:31.538795 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:31.538739 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:59:32.544654 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:32.544614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerStarted","Data":"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112"} Jul 07 16:59:33.204732 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:33.204695 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:33.205039 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:33.205011 2569 patch_prober.go:28] interesting pod/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:59:33.205108 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:33.205072 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 07 16:59:34.204271 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:34.204232 2569 patch_prober.go:28] interesting pod/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:59:34.204762 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:34.204293 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 07 16:59:35.205245 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:35.205194 2569 patch_prober.go:28] interesting pod/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:59:35.205729 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:35.205260 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 07 16:59:36.204779 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:36.204365 2569 patch_prober.go:28] interesting pod/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:59:36.204779 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:36.204424 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 07 16:59:37.208616 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:37.208588 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:37.209101 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:37.208947 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:37.209578 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:37.209561 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 16:59:37.351734 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:37.351707 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:37.451188 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:37.451158 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 16:59:43.832450 ip-10-0-142-74 kubenswrapper[2569]: I0707 16:59:43.832375 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:00.146882 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.146850 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724060-gngmd"] Jul 07 17:00:00.149631 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.149613 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4"] Jul 07 17:00:00.149754 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.149730 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:00.151994 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.151968 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:00.152222 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.152207 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-lb77l\"" Jul 07 17:00:00.160680 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.160652 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724060-gngmd"] Jul 07 17:00:00.163620 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.163591 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4"] Jul 07 17:00:00.196608 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.196585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt62j\" (UniqueName: \"kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j\") pod \"maas-api-key-cleanup-29724060-gngmd\" (UID: \"9efd8c50-80bd-4fc3-b91f-9379e4d54355\") " pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:00.196704 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.196627 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drztf\" (UniqueName: \"kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf\") pod \"maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4\" (UID: \"41ac0e89-432b-4abd-afee-35cdb597393a\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:00.297250 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.297227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drztf\" (UniqueName: \"kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf\") pod \"maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4\" (UID: \"41ac0e89-432b-4abd-afee-35cdb597393a\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:00.297338 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.297303 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt62j\" (UniqueName: \"kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j\") pod \"maas-api-key-cleanup-29724060-gngmd\" (UID: \"9efd8c50-80bd-4fc3-b91f-9379e4d54355\") " pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:00.307237 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.307216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt62j\" (UniqueName: \"kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j\") pod \"maas-api-key-cleanup-29724060-gngmd\" (UID: \"9efd8c50-80bd-4fc3-b91f-9379e4d54355\") " pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:00.307319 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.307216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drztf\" (UniqueName: \"kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf\") pod \"maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4\" (UID: \"41ac0e89-432b-4abd-afee-35cdb597393a\") " pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:00.464759 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.464697 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:00.474767 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.474744 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:00.617188 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.617166 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724060-gngmd"] Jul 07 17:00:00.619803 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:00:00.619770 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efd8c50_80bd_4fc3_b91f_9379e4d54355.slice/crio-20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4 WatchSource:0}: Error finding container 20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4: Status 404 returned error can't find the container with id 20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4 Jul 07 17:00:00.633856 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.633816 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4"] Jul 07 17:00:00.634737 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:00:00.634715 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ac0e89_432b_4abd_afee_35cdb597393a.slice/crio-47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f WatchSource:0}: Error finding container 47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f: Status 404 returned error can't find the container with id 47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f Jul 07 17:00:00.700988 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.700963 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" event={"ID":"9efd8c50-80bd-4fc3-b91f-9379e4d54355","Type":"ContainerStarted","Data":"20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4"} Jul 07 17:00:00.702145 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:00.702126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" event={"ID":"41ac0e89-432b-4abd-afee-35cdb597393a","Type":"ContainerStarted","Data":"47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f"} Jul 07 17:00:01.709066 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:01.709039 2569 generic.go:356] "Generic (PLEG): container finished" podID="9efd8c50-80bd-4fc3-b91f-9379e4d54355" containerID="f53c15162d0dbe2a3d4a85b861f9abd5dc356b63ebd955ec48cec56a5ab35cb8" exitCode=0 Jul 07 17:00:01.709426 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:01.709102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" event={"ID":"9efd8c50-80bd-4fc3-b91f-9379e4d54355","Type":"ContainerDied","Data":"f53c15162d0dbe2a3d4a85b861f9abd5dc356b63ebd955ec48cec56a5ab35cb8"} Jul 07 17:00:01.710694 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:01.710672 2569 generic.go:356] "Generic (PLEG): container finished" podID="41ac0e89-432b-4abd-afee-35cdb597393a" containerID="05b96f92ecaae672a4585640c712c6d722dbaf71160aaeb197014109849b45d1" exitCode=0 Jul 07 17:00:01.710800 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:01.710752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" event={"ID":"41ac0e89-432b-4abd-afee-35cdb597393a","Type":"ContainerDied","Data":"05b96f92ecaae672a4585640c712c6d722dbaf71160aaeb197014109849b45d1"} Jul 07 17:00:02.855270 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:02.855247 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:02.883199 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:02.883180 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:02.918823 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:02.918802 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt62j\" (UniqueName: \"kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j\") pod \"9efd8c50-80bd-4fc3-b91f-9379e4d54355\" (UID: \"9efd8c50-80bd-4fc3-b91f-9379e4d54355\") " Jul 07 17:00:02.920775 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:02.920750 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j" (OuterVolumeSpecName: "kube-api-access-bt62j") pod "9efd8c50-80bd-4fc3-b91f-9379e4d54355" (UID: "9efd8c50-80bd-4fc3-b91f-9379e4d54355"). InnerVolumeSpecName "kube-api-access-bt62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:00:03.019879 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.019803 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drztf\" (UniqueName: \"kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf\") pod \"41ac0e89-432b-4abd-afee-35cdb597393a\" (UID: \"41ac0e89-432b-4abd-afee-35cdb597393a\") " Jul 07 17:00:03.020104 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.020084 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt62j\" (UniqueName: \"kubernetes.io/projected/9efd8c50-80bd-4fc3-b91f-9379e4d54355-kube-api-access-bt62j\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:00:03.021853 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.021820 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf" (OuterVolumeSpecName: "kube-api-access-drztf") pod "41ac0e89-432b-4abd-afee-35cdb597393a" (UID: "41ac0e89-432b-4abd-afee-35cdb597393a"). InnerVolumeSpecName "kube-api-access-drztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:00:03.052675 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.052651 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:03.121017 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.120995 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drztf\" (UniqueName: \"kubernetes.io/projected/41ac0e89-432b-4abd-afee-35cdb597393a-kube-api-access-drztf\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:00:03.357662 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.357633 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:00:03.358046 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358032 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ac0e89-432b-4abd-afee-35cdb597393a" containerName="cleanup" Jul 07 17:00:03.358116 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358047 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ac0e89-432b-4abd-afee-35cdb597393a" containerName="cleanup" Jul 07 17:00:03.358116 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358073 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efd8c50-80bd-4fc3-b91f-9379e4d54355" containerName="cleanup" Jul 07 17:00:03.358116 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358079 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efd8c50-80bd-4fc3-b91f-9379e4d54355" containerName="cleanup" Jul 07 17:00:03.358275 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358143 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efd8c50-80bd-4fc3-b91f-9379e4d54355" containerName="cleanup" Jul 07 17:00:03.358275 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.358152 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ac0e89-432b-4abd-afee-35cdb597393a" containerName="cleanup" Jul 07 17:00:03.360361 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.360343 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.380677 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:00:03.380640 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.380677 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:00:03.380661 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-reuse-37478f-openshift-default-dockercfg-fj887\" is forbidden: User \"system:node:ip-10-0-142-74.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-74.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-reuse-37478f-openshift-default-dockercfg-fj887\"" type="*v1.Secret" Jul 07 17:00:03.424234 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424206 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424319 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424319 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424401 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424439 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424413 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5w6\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424508 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424545 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424579 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424544 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.424579 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.424568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.435126 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.435104 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:00:03.516699 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.516670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:03.525432 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525541 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525541 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525541 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525541 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525512 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525720 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5w6\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525982 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525804 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525982 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.525982 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.525928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.526145 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.526127 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.526449 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.526431 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.527736 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.527719 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.528057 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.528040 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.547493 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.547470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.549703 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.549684 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5w6\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6\") pod \"e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:03.723105 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.723054 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" Jul 07 17:00:03.723198 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.723101 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724060-gngmd" event={"ID":"9efd8c50-80bd-4fc3-b91f-9379e4d54355","Type":"ContainerDied","Data":"20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4"} Jul 07 17:00:03.723198 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.723142 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20500d1a3ebe355bb72d153caf129a69fd27b4f35b2ed2905f67b75513fa2ae4" Jul 07 17:00:03.724524 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.724509 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" Jul 07 17:00:03.724599 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.724544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4" event={"ID":"41ac0e89-432b-4abd-afee-35cdb597393a","Type":"ContainerDied","Data":"47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f"} Jul 07 17:00:03.724599 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.724578 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47244dc2a1f6fec3ff4a14ae7c52f80693dfb7af9320f06ed6dedfe2e8b72b2f" Jul 07 17:00:03.918588 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:03.918566 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:04.340337 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.340310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-reuse-37478f-openshift-default-dockercfg-fj887\"" Jul 07 17:00:04.341335 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.341320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:04.499951 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.499926 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:00:04.500880 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:00:04.500857 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cbb0b3_3d4c_4136_991c_6c54b07bbfc2.slice/crio-fdc65e750dc8990cd4ef77346c9b9ab5d400f619236b9256e9229a9a632b8c2f WatchSource:0}: Error finding container fdc65e750dc8990cd4ef77346c9b9ab5d400f619236b9256e9229a9a632b8c2f: Status 404 returned error can't find the container with id fdc65e750dc8990cd4ef77346c9b9ab5d400f619236b9256e9229a9a632b8c2f Jul 07 17:00:04.503019 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.502988 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:04.503096 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.503048 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:04.503096 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.503076 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:04.733974 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.733879 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerStarted","Data":"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88"} Jul 07 17:00:04.733974 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.733934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerStarted","Data":"fdc65e750dc8990cd4ef77346c9b9ab5d400f619236b9256e9229a9a632b8c2f"} Jul 07 17:00:04.760222 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:04.760175 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podStartSLOduration=1.760155928 podStartE2EDuration="1.760155928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:00:04.757223099 +0000 UTC m=+2209.066062378" watchObservedRunningTime="2026-07-07 17:00:04.760155928 +0000 UTC m=+2209.068995206" Jul 07 17:00:05.341762 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:05.341725 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:05.347138 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:05.347115 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:05.739638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:05.739569 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:05.740491 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:05.740475 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:06.822366 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:06.822323 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:06.957896 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:06.957845 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:07.077721 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:07.077615 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:07.388197 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:07.388170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:07.834359 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:07.834328 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:07.933362 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:07.933335 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:11.270113 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:11.270077 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:16.749015 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:16.748974 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:20.740961 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.740925 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 07 17:00:20.741375 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.740996 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 07 17:00:20.843710 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.843677 2569 generic.go:356] "Generic (PLEG): container finished" podID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerID="8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88" exitCode=0 Jul 07 17:00:20.843889 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.843746 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerDied","Data":"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88"} Jul 07 17:00:20.844163 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.844144 2569 scope.go:117] "RemoveContainer" containerID="8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88" Jul 07 17:00:20.845280 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.845250 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:20.845357 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.845339 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:20.845400 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:20.845384 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:00:21.851873 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:21.851825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerStarted","Data":"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1"} Jul 07 17:00:22.342323 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:22.342290 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:22.342519 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:22.342433 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 07 17:00:22.342519 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:22.342474 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 07 17:00:23.342385 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:23.342349 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 07 17:00:23.342788 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:23.342400 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 07 17:00:24.341993 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:24.341957 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 07 17:00:24.342204 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:24.342034 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 07 17:00:25.341955 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:25.341916 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 07 17:00:25.342354 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:25.341989 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 07 17:00:26.346172 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:26.346144 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:26.346623 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:26.346601 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:26.347329 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:26.347308 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:00:48.708727 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:48.708648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:49.985975 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:49.985933 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:00:54.105848 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:00:54.105814 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:25.534564 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:25.534532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:30.308255 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:30.308220 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:31.666485 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:31.666450 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:37.527662 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:37.527628 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:52.226211 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.226173 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:52.389988 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.389950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:52.531098 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.531030 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:01:52.531254 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.531226 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" containerID="cri-o://4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1" gracePeriod=30 Jul 07 17:01:52.590893 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.590867 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:52.824476 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:52.824396 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:53.770179 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:53.770149 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:54.769889 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:54.769817 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:54.956500 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:54.956472 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:55.371949 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.371920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:55.489541 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.489507 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:55.605167 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.605141 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:55.769486 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.769410 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:01:55.806380 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.806351 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4"] Jul 07 17:01:55.809770 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:55.809750 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-inf-a-1b64ca-29724060-xbxh4"] Jul 07 17:01:56.268491 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:56.268460 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ac0e89-432b-4abd-afee-35cdb597393a" path="/var/lib/kubelet/pods/41ac0e89-432b-4abd-afee-35cdb597393a/volumes" Jul 07 17:01:56.348472 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:56.348445 2569 patch_prober.go:28] interesting pod/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 17:01:56.348617 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:56.348493 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 17:01:57.778613 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.778589 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:01:57.852237 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852213 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s5w6\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852368 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852260 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852368 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852303 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852368 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852338 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852535 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852382 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852535 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852418 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852535 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852460 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852535 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852493 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852543 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert\") pod \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\" (UID: \"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2\") " Jul 07 17:01:57.852732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852618 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:01:57.852732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852637 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:01:57.852891 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852871 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:01:57.852974 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.852956 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-credential-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.853016 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.853002 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.853065 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.853017 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-workload-certs\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.853132 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.853106 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:01:57.853169 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.853151 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:01:57.854333 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.854313 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6" (OuterVolumeSpecName: "kube-api-access-4s5w6") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "kube-api-access-4s5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:01:57.854883 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.854862 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:01:57.854962 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.854941 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:01:57.855002 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.854948 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" (UID: "80cbb0b3-3d4c-4136-991c-6c54b07bbfc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:01:57.954180 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954155 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istiod-ca-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.954180 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954178 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5w6\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-kube-api-access-4s5w6\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.954313 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954192 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-data\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.954313 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954204 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-token\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.954313 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954214 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-envoy\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:57.954313 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:57.954226 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2-istio-podinfo\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:01:58.405534 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.405507 2569 generic.go:356] "Generic (PLEG): container finished" podID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerID="4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1" exitCode=0 Jul 07 17:01:58.405644 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.405557 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" Jul 07 17:01:58.405644 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.405594 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerDied","Data":"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1"} Jul 07 17:01:58.405726 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.405645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl" event={"ID":"80cbb0b3-3d4c-4136-991c-6c54b07bbfc2","Type":"ContainerDied","Data":"fdc65e750dc8990cd4ef77346c9b9ab5d400f619236b9256e9229a9a632b8c2f"} Jul 07 17:01:58.405726 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.405670 2569 scope.go:117] "RemoveContainer" containerID="4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1" Jul 07 17:01:58.415709 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.415677 2569 scope.go:117] "RemoveContainer" containerID="8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88" Jul 07 17:01:58.424679 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.424663 2569 scope.go:117] "RemoveContainer" containerID="4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1" Jul 07 17:01:58.424972 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:01:58.424954 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1\": container with ID starting with 4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1 not found: ID does not exist" containerID="4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1" Jul 07 17:01:58.425034 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.424978 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1"} err="failed to get container status \"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1\": rpc error: code = NotFound desc = could not find container \"4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1\": container with ID starting with 4235b76590376dad533f9efddbee3a3fc833eebd9208e33dae4fba167c4e63c1 not found: ID does not exist" Jul 07 17:01:58.425034 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.425000 2569 scope.go:117] "RemoveContainer" containerID="8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88" Jul 07 17:01:58.425258 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:01:58.425236 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88\": container with ID starting with 8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88 not found: ID does not exist" containerID="8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88" Jul 07 17:01:58.425317 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.425269 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88"} err="failed to get container status \"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88\": rpc error: code = NotFound desc = could not find container \"8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88\": container with ID starting with 8ac4d358ba1c7913977e5ba152a7125c5e72f6abd426222ff8ef4452366c0b88 not found: ID does not exist" Jul 07 17:01:58.435953 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.435934 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:01:58.438284 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:58.438263 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-reuse-37478f-openshift-default-d7fbb8797-rpbbl"] Jul 07 17:01:59.919208 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:01:59.919170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:00.029024 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:00.028988 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:00.268581 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:00.268505 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" path="/var/lib/kubelet/pods/80cbb0b3-3d4c-4136-991c-6c54b07bbfc2/volumes" Jul 07 17:02:00.442415 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:00.442387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:37.589097 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:37.589057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:37.906343 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:37.906259 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:38.142165 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:38.142128 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:38.451134 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:38.451102 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:38.903873 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:38.903829 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:38.996294 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:38.996266 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:39.221783 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:39.221704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:44.883732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:44.883698 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:44.983525 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:44.983495 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:45.156732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:45.156650 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 17:02:45.156950 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:45.156924 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" containerID="cri-o://1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112" gracePeriod=30 Jul 07 17:02:45.173873 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:45.173829 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:45.424912 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:45.424808 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:45.799920 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:45.799810 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:46.148012 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:46.147983 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:46.245099 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:46.245072 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:48.810291 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:48.810249 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:48.975359 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:48.975332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:49.206936 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:49.206909 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:49.681948 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:49.681914 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:49.787775 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:49.787748 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:50.409704 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.409681 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 17:02:50.479521 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479454 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479521 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479485 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479521 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479506 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqxrc\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479526 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479541 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479573 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479603 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.479781 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479712 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs\") pod \"246866d5-c9f3-41a7-83c6-86de36010631\" (UID: \"246866d5-c9f3-41a7-83c6-86de36010631\") " Jul 07 17:02:50.480136 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479762 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:02:50.480136 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.480042 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:02:50.480136 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.479981 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:02:50.480136 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.480113 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-credential-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.480356 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.480238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:02:50.480437 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.480415 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data" (OuterVolumeSpecName: "istio-data") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:02:50.481670 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.481645 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:02:50.481772 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.481724 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token" (OuterVolumeSpecName: "istio-token") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:02:50.482274 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.482249 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc" (OuterVolumeSpecName: "kube-api-access-qqxrc") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "kube-api-access-qqxrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:02:50.482341 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.482262 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "246866d5-c9f3-41a7-83c6-86de36010631" (UID: "246866d5-c9f3-41a7-83c6-86de36010631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:02:50.580471 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580442 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246866d5-c9f3-41a7-83c6-86de36010631-istiod-ca-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580471 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580468 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxrc\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-kube-api-access-qqxrc\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580484 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246866d5-c9f3-41a7-83c6-86de36010631-istio-podinfo\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580509 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-envoy\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580523 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-istio-data\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580536 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246866d5-c9f3-41a7-83c6-86de36010631-istio-token\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580549 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.580638 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.580564 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246866d5-c9f3-41a7-83c6-86de36010631-workload-certs\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:02:50.708813 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.708783 2569 generic.go:356] "Generic (PLEG): container finished" podID="246866d5-c9f3-41a7-83c6-86de36010631" containerID="1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112" exitCode=0 Jul 07 17:02:50.708936 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.708858 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" Jul 07 17:02:50.708936 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.708875 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerDied","Data":"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112"} Jul 07 17:02:50.709014 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.708941 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk" event={"ID":"246866d5-c9f3-41a7-83c6-86de36010631","Type":"ContainerDied","Data":"66d11c426fbc996b0e9e5167266d5646da70e87847eb4e632d34875603a422f4"} Jul 07 17:02:50.709014 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.708969 2569 scope.go:117] "RemoveContainer" containerID="1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112" Jul 07 17:02:50.718333 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.718316 2569 scope.go:117] "RemoveContainer" containerID="03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00" Jul 07 17:02:50.726789 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.726764 2569 scope.go:117] "RemoveContainer" containerID="1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112" Jul 07 17:02:50.727052 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:02:50.727034 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112\": container with ID starting with 1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112 not found: ID does not exist" containerID="1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112" Jul 07 17:02:50.727110 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.727058 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112"} err="failed to get container status \"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112\": rpc error: code = NotFound desc = could not find container \"1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112\": container with ID starting with 1ad7c83692c48fb138a7bff87204ad581f792ba5dd00d148fb6dc3382097f112 not found: ID does not exist" Jul 07 17:02:50.727110 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.727075 2569 scope.go:117] "RemoveContainer" containerID="03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00" Jul 07 17:02:50.727282 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:02:50.727267 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00\": container with ID starting with 03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00 not found: ID does not exist" containerID="03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00" Jul 07 17:02:50.727324 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.727285 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00"} err="failed to get container status \"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00\": rpc error: code = NotFound desc = could not find container \"03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00\": container with ID starting with 03fe60a3f7eacad29b5a10721746efc0fd2f615527540e54f3981d9e391c5d00 not found: ID does not exist" Jul 07 17:02:50.734302 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.734242 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 17:02:50.741150 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:50.741123 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4b5d6a-openshift-default-7b7f9f5c55-8qjxk"] Jul 07 17:02:52.268150 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:52.268116 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246866d5-c9f3-41a7-83c6-86de36010631" path="/var/lib/kubelet/pods/246866d5-c9f3-41a7-83c6-86de36010631/volumes" Jul 07 17:02:53.365647 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:53.365612 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:53.871626 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:53.871597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:54.248153 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248087 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:02:54.248496 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248484 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:02:54.248539 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248499 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:02:54.248539 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248515 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.248539 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248521 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.248634 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248540 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:02:54.248634 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248546 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:02:54.248634 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.248634 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248565 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.248634 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248630 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:02:54.248783 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248639 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.248783 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.248648 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cbb0b3-3d4c-4136-991c-6c54b07bbfc2" containerName="istio-proxy" Jul 07 17:02:54.251748 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.251726 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.254336 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.254311 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 17:02:54.254528 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.254508 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4f1f9c-openshift-default-dockercfg-9cjzv\"" Jul 07 17:02:54.268703 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.268681 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:02:54.294991 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.294972 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:54.312161 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312264 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbg88\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312337 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312337 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312411 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312360 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312411 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312390 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312476 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312518 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312481 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.312554 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.312532 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.392663 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.392635 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:54.413745 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.413857 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.413857 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbg88\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.413857 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.413857 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414086 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414086 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414086 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414086 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.413998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414298 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.414163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414298 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.414216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414410 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.414346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414466 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.414436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.414783 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.414758 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.416129 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.416109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.416466 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.416446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.422380 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.422357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.423121 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.423097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbg88\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88\") pod \"e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.562029 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.561912 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:54.715576 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.715541 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:02:54.716952 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:02:54.716924 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afd7834_3a57_46d5_a0b5_dc19cbf8480e.slice/crio-1110bcbba089848ef5db38d2c4e0543357f9aa243bcbe83ff3c81f374255eedb WatchSource:0}: Error finding container 1110bcbba089848ef5db38d2c4e0543357f9aa243bcbe83ff3c81f374255eedb: Status 404 returned error can't find the container with id 1110bcbba089848ef5db38d2c4e0543357f9aa243bcbe83ff3c81f374255eedb Jul 07 17:02:54.719121 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.719093 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:02:54.719204 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.719166 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:02:54.719272 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.719217 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:02:54.734741 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:54.734708 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerStarted","Data":"1110bcbba089848ef5db38d2c4e0543357f9aa243bcbe83ff3c81f374255eedb"} Jul 07 17:02:55.742159 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:55.742118 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerStarted","Data":"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2"} Jul 07 17:02:55.769286 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:55.769237 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podStartSLOduration=1.769222158 podStartE2EDuration="1.769222158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:02:55.766798323 +0000 UTC m=+2380.075637601" watchObservedRunningTime="2026-07-07 17:02:55.769222158 +0000 UTC m=+2380.078061470" Jul 07 17:02:56.139360 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:56.139331 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:56.562755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:56.562676 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:56.567292 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:56.567263 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:56.748635 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:56.748605 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:56.749393 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:56.749376 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:02:57.785980 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:57.785949 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:57.887698 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:57.887670 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:57.961964 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:57.961938 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:58.143682 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:58.143657 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:58.445087 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:58.445020 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:02:58.541988 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:02:58.541966 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:02.300715 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:02.300679 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:10.526607 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:10.526574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:10.626002 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:10.625972 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:10.818013 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:10.817934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:10.926821 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:10.926783 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:16.362969 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.362945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 17:03:16.374882 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.374863 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 17:03:16.866703 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.866674 2569 generic.go:356] "Generic (PLEG): container finished" podID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerID="4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2" exitCode=0 Jul 07 17:03:16.866881 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.866742 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerDied","Data":"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2"} Jul 07 17:03:16.867133 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.867116 2569 scope.go:117] "RemoveContainer" containerID="4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2" Jul 07 17:03:16.868129 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.868098 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:03:16.868209 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.868194 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:03:16.868253 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:16.868241 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 17:03:17.135535 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:17.135459 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:17.254619 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:17.254584 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:17.846339 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:17.846299 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:17.875858 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:17.875807 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerStarted","Data":"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729"} Jul 07 17:03:18.089276 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.089243 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:18.347749 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.347721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:18.446905 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.446866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:18.548004 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.547970 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:18.562095 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.562069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:18.562278 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.562257 2569 patch_prober.go:28] interesting pod/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 17:03:18.562341 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.562321 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 17:03:18.793621 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:18.793532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:19.563113 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:19.563079 2569 patch_prober.go:28] interesting pod/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 17:03:19.563478 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:19.563162 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 17:03:20.562266 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:20.562231 2569 patch_prober.go:28] interesting pod/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 17:03:20.562435 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:20.562288 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 17:03:21.562780 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:21.562747 2569 patch_prober.go:28] interesting pod/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 07 17:03:21.563186 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:21.562796 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 07 17:03:22.566682 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:22.566652 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:22.567117 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:22.566909 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:22.567667 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:22.567644 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:24.559079 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:24.559009 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:24.633361 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:24.633332 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:24.829611 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:24.829540 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:03:24.851732 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:24.851705 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:24.933418 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:24.933381 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" containerID="cri-o://4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729" gracePeriod=30 Jul 07 17:03:25.048124 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:25.048092 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:25.404440 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:25.404411 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:25.741608 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:25.741517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:25.960142 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:25.960108 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:28.996647 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:28.996617 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:29.126078 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:29.126049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:29.202602 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:29.202574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:29.719680 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:29.719647 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:29.817022 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:29.816989 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:30.194458 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.194431 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:30.318581 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318516 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318581 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318565 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318778 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318597 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318778 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318626 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318778 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318652 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbg88\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318778 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.318778 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318731 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.319072 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318823 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.319072 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.318862 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert\") pod \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\" (UID: \"2afd7834-3a57-46d5-a0b5-dc19cbf8480e\") " Jul 07 17:03:30.319184 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.319112 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:03:30.319245 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.319208 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:03:30.319352 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.319328 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:03:30.319483 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.319451 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:03:30.319657 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.319627 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:03:30.321254 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.321229 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:03:30.321371 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.321320 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:03:30.321598 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.321518 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88" (OuterVolumeSpecName: "kube-api-access-kbg88") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "kube-api-access-kbg88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:03:30.321680 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.321609 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2afd7834-3a57-46d5-a0b5-dc19cbf8480e" (UID: "2afd7834-3a57-46d5-a0b5-dc19cbf8480e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 17:03:30.420481 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420452 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420484 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-credential-socket\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420500 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istiod-ca-cert\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420513 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-envoy\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420525 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-data\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420533 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-podinfo\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420544 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-istio-token\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420559 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbg88\" (UniqueName: \"kubernetes.io/projected/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-kube-api-access-kbg88\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.420594 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.420574 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afd7834-3a57-46d5-a0b5-dc19cbf8480e-workload-certs\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:03:30.968876 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.968825 2569 generic.go:356] "Generic (PLEG): container finished" podID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerID="4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729" exitCode=0 Jul 07 17:03:30.969045 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.968896 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" Jul 07 17:03:30.969045 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.968939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerDied","Data":"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729"} Jul 07 17:03:30.969045 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.969004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8" event={"ID":"2afd7834-3a57-46d5-a0b5-dc19cbf8480e","Type":"ContainerDied","Data":"1110bcbba089848ef5db38d2c4e0543357f9aa243bcbe83ff3c81f374255eedb"} Jul 07 17:03:30.969045 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.969030 2569 scope.go:117] "RemoveContainer" containerID="4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729" Jul 07 17:03:30.986101 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.986081 2569 scope.go:117] "RemoveContainer" containerID="4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2" Jul 07 17:03:30.994724 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.994707 2569 scope.go:117] "RemoveContainer" containerID="4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729" Jul 07 17:03:30.994989 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:03:30.994968 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729\": container with ID starting with 4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729 not found: ID does not exist" containerID="4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729" Jul 07 17:03:30.995066 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.995000 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729"} err="failed to get container status \"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729\": rpc error: code = NotFound desc = could not find container \"4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729\": container with ID starting with 4f7b32834e6c2d3058474990915294f42dc46caca4555d04eb405e3d93b82729 not found: ID does not exist" Jul 07 17:03:30.995066 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.995022 2569 scope.go:117] "RemoveContainer" containerID="4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2" Jul 07 17:03:30.995285 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:03:30.995266 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2\": container with ID starting with 4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2 not found: ID does not exist" containerID="4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2" Jul 07 17:03:30.995324 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:30.995292 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2"} err="failed to get container status \"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2\": rpc error: code = NotFound desc = could not find container \"4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2\": container with ID starting with 4b3803560e374dc57f791535d6bf963313d7c270a6d3932209a516c2709847c2 not found: ID does not exist" Jul 07 17:03:31.000611 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:31.000587 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:03:31.010217 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:31.010191 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4f1f9c-openshift-default-55658fbf79-mcfg8"] Jul 07 17:03:32.267230 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:32.267197 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" path="/var/lib/kubelet/pods/2afd7834-3a57-46d5-a0b5-dc19cbf8480e/volumes" Jul 07 17:03:33.473478 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:33.473442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:33.834434 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:33.834405 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:34.170246 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:34.170151 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:34.361922 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:34.361896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:36.997520 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:36.997486 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:37.108886 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:37.108847 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:37.205533 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:37.205504 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:37.304753 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:37.304688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:37.653825 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:37.653801 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:37.748370 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:37.748346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:38.057427 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:38.057324 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:41.505941 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:41.505905 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:50.578418 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:50.578379 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:50.676298 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:50.676272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:58.439109 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:58.439078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:58.543755 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:58.543726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:03:58.639878 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:03:58.639853 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:04:21.419855 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:04:21.419809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:04:25.065316 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:04:25.065284 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:05:50.804960 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:05:50.804930 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:05:52.117395 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:05:52.117363 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:06:16.473109 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:16.473075 2569 scope.go:117] "RemoveContainer" containerID="05b96f92ecaae672a4585640c712c6d722dbaf71160aaeb197014109849b45d1" Jul 07 17:06:19.162139 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:19.162102 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:06:20.122094 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:20.122057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:06:29.916488 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.916450 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:06:29.917421 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917386 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:06:29.917421 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917418 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:06:29.917588 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917452 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:06:29.917588 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917461 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:06:29.917695 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917632 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:06:29.917695 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.917656 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="246866d5-c9f3-41a7-83c6-86de36010631" containerName="istio-proxy" Jul 07 17:06:29.920775 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.920742 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:29.927648 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.927554 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"kube-root-ca.crt\"" Jul 07 17:06:29.927737 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.927689 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"default-dockercfg-fzgk9\"" Jul 07 17:06:29.927881 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.927862 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"sub-test-model-58e7b2-kserve-self-signed-certs\"" Jul 07 17:06:29.927881 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.927875 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-58e7b2\"/\"openshift-service-ca.crt\"" Jul 07 17:06:29.944436 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.944416 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:06:29.951371 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:29.951352 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:06:30.021139 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.021330 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7cn\" (UniqueName: \"kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.021330 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.021330 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.021496 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021360 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.021496 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.021387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122139 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122139 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122370 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122370 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122370 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7cn\" (UniqueName: \"kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122370 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122595 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122655 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122625 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.122886 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.122868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.124618 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.124593 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.124724 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.124692 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.132527 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.132507 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7cn\" (UniqueName: \"kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn\") pod \"sub-test-model-58e7b2-kserve-fcd64d78f-bl74v\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.230005 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.229949 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:30.373819 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.373786 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:06:30.374760 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:06:30.374733 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8b814b0_bbf4_4eed_a83a_3cf9e7170cef.slice/crio-fc187e96f89181255ee08ebc3f5608071a6d7ed6c74e9695597cf11cf4cb010c WatchSource:0}: Error finding container fc187e96f89181255ee08ebc3f5608071a6d7ed6c74e9695597cf11cf4cb010c: Status 404 returned error can't find the container with id fc187e96f89181255ee08ebc3f5608071a6d7ed6c74e9695597cf11cf4cb010c Jul 07 17:06:30.989637 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.989602 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" event={"ID":"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef","Type":"ContainerStarted","Data":"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984"} Jul 07 17:06:30.990041 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:30.989651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" event={"ID":"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef","Type":"ContainerStarted","Data":"fc187e96f89181255ee08ebc3f5608071a6d7ed6c74e9695597cf11cf4cb010c"} Jul 07 17:06:31.015951 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:31.015896 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" podStartSLOduration=2.015880841 podStartE2EDuration="2.015880841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:06:31.014070438 +0000 UTC m=+2595.322909715" watchObservedRunningTime="2026-07-07 17:06:31.015880841 +0000 UTC m=+2595.324720118" Jul 07 17:06:40.230579 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:40.230497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:40.230579 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:40.230546 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:40.243366 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:40.243342 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:06:41.058264 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:06:41.058228 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:07:00.230803 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:00.230766 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:07:42.860489 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:42.860459 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:07:42.895877 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:42.895816 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:07:43.634261 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:43.634230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:07:57.879894 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:57.879800 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" podUID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" containerName="main" containerID="cri-o://cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984" gracePeriod=45 Jul 07 17:07:58.130605 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.130548 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:07:58.218596 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218569 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218738 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218611 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218738 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218648 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218738 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218681 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218738 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218716 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218985 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218773 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb7cn\" (UniqueName: \"kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn\") pod \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\" (UID: \"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef\") " Jul 07 17:07:58.218985 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218957 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:07:58.218985 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.218965 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home" (OuterVolumeSpecName: "home") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:07:58.219127 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.219016 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:07:58.219186 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.219162 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tmp-dir\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.219186 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.219181 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-home\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.219282 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.219190 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-model-cache\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.220853 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.220803 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:07:58.220966 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.220862 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm" (OuterVolumeSpecName: "dshm") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:07:58.220966 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.220890 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn" (OuterVolumeSpecName: "kube-api-access-qb7cn") pod "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" (UID: "e8b814b0-bbf4-4eed-a83a-3cf9e7170cef"). InnerVolumeSpecName "kube-api-access-qb7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:07:58.320361 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.320339 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-tls-certs\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.320361 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.320360 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-dshm\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.320490 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.320370 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7cn\" (UniqueName: \"kubernetes.io/projected/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef-kube-api-access-qb7cn\") on node \"ip-10-0-142-74.ec2.internal\" DevicePath \"\"" Jul 07 17:07:58.487423 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.487359 2569 generic.go:356] "Generic (PLEG): container finished" podID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" containerID="cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984" exitCode=0 Jul 07 17:07:58.487423 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.487401 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" Jul 07 17:07:58.487545 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.487402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" event={"ID":"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef","Type":"ContainerDied","Data":"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984"} Jul 07 17:07:58.487545 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.487514 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v" event={"ID":"e8b814b0-bbf4-4eed-a83a-3cf9e7170cef","Type":"ContainerDied","Data":"fc187e96f89181255ee08ebc3f5608071a6d7ed6c74e9695597cf11cf4cb010c"} Jul 07 17:07:58.487545 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.487538 2569 scope.go:117] "RemoveContainer" containerID="cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984" Jul 07 17:07:58.500862 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.500827 2569 scope.go:117] "RemoveContainer" containerID="cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984" Jul 07 17:07:58.501170 ip-10-0-142-74 kubenswrapper[2569]: E0707 17:07:58.501147 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984\": container with ID starting with cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984 not found: ID does not exist" containerID="cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984" Jul 07 17:07:58.501226 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.501182 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984"} err="failed to get container status \"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984\": rpc error: code = NotFound desc = could not find container \"cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984\": container with ID starting with cc7494f1c27ae5740a7068094256876a2c5138a9218ab49d36648988c8ed5984 not found: ID does not exist" Jul 07 17:07:58.510462 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.510439 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:07:58.516588 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:07:58.516567 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-58e7b2/sub-test-model-58e7b2-kserve-fcd64d78f-bl74v"] Jul 07 17:08:00.268980 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:00.268941 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" path="/var/lib/kubelet/pods/e8b814b0-bbf4-4eed-a83a-3cf9e7170cef/volumes" Jul 07 17:08:14.861051 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:14.861015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:08:16.400511 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:16.400484 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 17:08:16.413050 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:16.413029 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 17:08:39.926665 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:39.926625 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:08:51.618633 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:51.618588 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:08:51.737165 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:51.737120 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:08:59.465872 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:59.465809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:08:59.569291 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:08:59.569260 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:21.094012 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:21.093974 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:22.262275 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:22.262235 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:27.913381 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:27.913339 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:28.305597 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:28.305525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:28.405545 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:28.405514 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:28.508554 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:28.508532 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:28.658047 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:28.658022 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:33.374204 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:33.374166 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:34.037520 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:34.037489 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:34.156342 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:34.156312 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:39.622567 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:39.622485 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:39.945736 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:39.945630 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:40.270220 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:40.270103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:40.453423 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:40.453388 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:40.819293 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:40.819266 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:40.919890 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:40.919863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:41.017769 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:41.017745 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:41.167369 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:41.167329 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:47.626866 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:47.626815 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:47.727052 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:47.727014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:47.927130 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:47.927060 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:48.165435 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:48.165403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:09:48.602489 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:48.602465 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d5597fcd5-c472h_d2ee68c1-4d2a-4985-b6b8-c9960946ad7d/authorino/0.log" Jul 07 17:09:53.905700 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:53.905665 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724045-b6hdd_229e8f96-74fc-439e-b82c-8d4c45079524/cleanup/0.log" Jul 07 17:09:54.047259 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:54.047234 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724060-gngmd_9efd8c50-80bd-4fc3-b91f-9379e4d54355/cleanup/0.log" Jul 07 17:09:54.184534 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:54.184478 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56dc7d6c67-kcgvj_54193f19-612e-452b-8914-07912920c6b7/manager/0.log" Jul 07 17:09:54.871670 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:54.871640 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-cwsf9_ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766/copy-manifests/0.log" Jul 07 17:09:54.879843 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:54.879809 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-cwsf9_ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766/manager/0.log" Jul 07 17:09:55.009440 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.009419 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-n97dk_1d973308-f677-4de8-9a4d-de98aeefd85d/postgres/0.log" Jul 07 17:09:55.828398 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.828365 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/util/0.log" Jul 07 17:09:55.835891 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.835872 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/pull/0.log" Jul 07 17:09:55.843568 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.843533 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/extract/0.log" Jul 07 17:09:55.967650 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.967632 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/extract/0.log" Jul 07 17:09:55.974924 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.974901 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/util/0.log" Jul 07 17:09:55.982587 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:55.982565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/pull/0.log" Jul 07 17:09:56.114454 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.114433 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/util/0.log" Jul 07 17:09:56.122245 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.122228 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/pull/0.log" Jul 07 17:09:56.129962 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.129945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/extract/0.log" Jul 07 17:09:56.250960 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.250939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/util/0.log" Jul 07 17:09:56.258678 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.258657 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/pull/0.log" Jul 07 17:09:56.266349 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.266331 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/extract/0.log" Jul 07 17:09:56.389504 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.389442 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d5597fcd5-c472h_d2ee68c1-4d2a-4985-b6b8-c9960946ad7d/authorino/0.log" Jul 07 17:09:56.746914 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:56.746850 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-76nng_42993fa1-f26c-4226-9776-ffc7d9368d16/kuadrant-console-plugin/0.log" Jul 07 17:09:58.036707 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:58.036674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-mmjx5_9827d53c-ce10-4438-b9d5-ccd350c49338/kube-auth-proxy/0.log" Jul 07 17:09:58.311614 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:58.311539 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-zb5zb_7292ff97-1198-4bce-8f35-2c4da06d79e2/payload-pre-processing/0.log" Jul 07 17:09:58.442682 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:58.442653 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-84j4f_960c0798-0406-4a62-b695-bc3a4fef2020/payload-processing/0.log" Jul 07 17:09:59.224024 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:09:59.223992 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qvg6l_c2ec10fb-621e-4317-82a7-024d69e9ef00/main/0.log" Jul 07 17:10:04.103330 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.103297 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nkp5f/must-gather-2m8jx"] Jul 07 17:10:04.103717 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.103697 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" containerName="main" Jul 07 17:10:04.103717 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.103710 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" containerName="main" Jul 07 17:10:04.103789 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.103780 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afd7834-3a57-46d5-a0b5-dc19cbf8480e" containerName="istio-proxy" Jul 07 17:10:04.103823 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.103788 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b814b0-bbf4-4eed-a83a-3cf9e7170cef" containerName="main" Jul 07 17:10:04.107333 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.107315 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.110585 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.110562 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkp5f\"/\"openshift-service-ca.crt\"" Jul 07 17:10:04.110585 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.110563 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nkp5f\"/\"default-dockercfg-4vg9d\"" Jul 07 17:10:04.111185 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.111158 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nkp5f\"/\"kube-root-ca.crt\"" Jul 07 17:10:04.112728 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.112704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:10:04.122377 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.122353 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/must-gather-2m8jx"] Jul 07 17:10:04.189133 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.189114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d4959f5-7981-4d79-b66b-115f6077a10c-must-gather-output\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.189236 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.189174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6k4g\" (UniqueName: \"kubernetes.io/projected/4d4959f5-7981-4d79-b66b-115f6077a10c-kube-api-access-z6k4g\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.236960 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.236937 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" limit=1024 Jul 07 17:10:04.289642 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.289619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d4959f5-7981-4d79-b66b-115f6077a10c-must-gather-output\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.289754 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.289694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6k4g\" (UniqueName: \"kubernetes.io/projected/4d4959f5-7981-4d79-b66b-115f6077a10c-kube-api-access-z6k4g\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.289977 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.289957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4d4959f5-7981-4d79-b66b-115f6077a10c-must-gather-output\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.301423 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.301397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6k4g\" (UniqueName: \"kubernetes.io/projected/4d4959f5-7981-4d79-b66b-115f6077a10c-kube-api-access-z6k4g\") pod \"must-gather-2m8jx\" (UID: \"4d4959f5-7981-4d79-b66b-115f6077a10c\") " pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.417566 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.417509 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" Jul 07 17:10:04.552004 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.551973 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/must-gather-2m8jx"] Jul 07 17:10:04.553338 ip-10-0-142-74 kubenswrapper[2569]: W0707 17:10:04.553313 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4959f5_7981_4d79_b66b_115f6077a10c.slice/crio-c1423d067c9a7aded1cf41966cafd3af40a206f6a0f1f1eb60017ef45c01966e WatchSource:0}: Error finding container c1423d067c9a7aded1cf41966cafd3af40a206f6a0f1f1eb60017ef45c01966e: Status 404 returned error can't find the container with id c1423d067c9a7aded1cf41966cafd3af40a206f6a0f1f1eb60017ef45c01966e Jul 07 17:10:04.555192 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:04.555174 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:10:05.215669 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:05.215632 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" event={"ID":"4d4959f5-7981-4d79-b66b-115f6077a10c","Type":"ContainerStarted","Data":"c1423d067c9a7aded1cf41966cafd3af40a206f6a0f1f1eb60017ef45c01966e"} Jul 07 17:10:06.225069 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:06.224982 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" event={"ID":"4d4959f5-7981-4d79-b66b-115f6077a10c","Type":"ContainerStarted","Data":"2a2df828b2be557d81ae0ca3d12b96d8f3ebc26bcd5b7e3d581e47cb52695cd4"} Jul 07 17:10:06.225069 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:06.225036 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" event={"ID":"4d4959f5-7981-4d79-b66b-115f6077a10c","Type":"ContainerStarted","Data":"2761c7813c142db12125e9a21b4f80b5c568a7fff19576cd0ace173eed37a758"} Jul 07 17:10:06.954242 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:06.954204 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nk2sf_f6620e90-14c4-4008-a09e-d56578336e2a/global-pull-secret-syncer/0.log" Jul 07 17:10:07.064791 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:07.064762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ghsxc_34608c24-a6e1-4a49-9cc7-2f5bd7fe8b6b/konnectivity-agent/0.log" Jul 07 17:10:07.239382 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:07.239307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-74.ec2.internal_2dca9c09f5a6c3fe6fd4a6f6555384fa/haproxy/0.log" Jul 07 17:10:11.620671 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.620641 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/extract/0.log" Jul 07 17:10:11.657583 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.657557 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/util/0.log" Jul 07 17:10:11.691643 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.691613 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z6pst_f473e35c-7340-4b50-8293-b2004f9f7e89/pull/0.log" Jul 07 17:10:11.726082 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.726048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/extract/0.log" Jul 07 17:10:11.757858 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.757808 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/util/0.log" Jul 07 17:10:11.786525 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.786462 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0m4dfm_f2dcac0f-c709-4603-884d-617f73832558/pull/0.log" Jul 07 17:10:11.822246 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.822198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/extract/0.log" Jul 07 17:10:11.846702 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.846676 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/util/0.log" Jul 07 17:10:11.875047 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.874945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed734m6xp_be64e709-b1cb-4287-8bff-e6829a7e3134/pull/0.log" Jul 07 17:10:11.912242 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.912216 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/extract/0.log" Jul 07 17:10:11.947100 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.947063 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/util/0.log" Jul 07 17:10:11.988564 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:11.988514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16qp7m_f79b6b94-003f-4802-8ab4-8e4fe0c38eae/pull/0.log" Jul 07 17:10:12.350012 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:12.349984 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d5597fcd5-c472h_d2ee68c1-4d2a-4985-b6b8-c9960946ad7d/authorino/0.log" Jul 07 17:10:12.451305 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:12.451278 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-76nng_42993fa1-f26c-4226-9776-ffc7d9368d16/kuadrant-console-plugin/0.log" Jul 07 17:10:14.476209 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.476177 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/alertmanager/0.log" Jul 07 17:10:14.503123 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.503088 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/config-reloader/0.log" Jul 07 17:10:14.531490 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.531448 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/kube-rbac-proxy-web/0.log" Jul 07 17:10:14.562009 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.561978 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/kube-rbac-proxy/0.log" Jul 07 17:10:14.591720 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.591661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/kube-rbac-proxy-metric/0.log" Jul 07 17:10:14.616227 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.616198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/prom-label-proxy/0.log" Jul 07 17:10:14.645731 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.645696 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5bcb872a-4a05-4343-b439-c14f71d5e2b1/init-config-reloader/0.log" Jul 07 17:10:14.742415 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.742384 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tlnrc_1197d210-3afa-4053-a758-ff0ce4f8e52f/kube-state-metrics/0.log" Jul 07 17:10:14.770011 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.769978 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tlnrc_1197d210-3afa-4053-a758-ff0ce4f8e52f/kube-rbac-proxy-main/0.log" Jul 07 17:10:14.804556 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.804525 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tlnrc_1197d210-3afa-4053-a758-ff0ce4f8e52f/kube-rbac-proxy-self/0.log" Jul 07 17:10:14.915282 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.915256 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtq8q_052e5cc4-8625-46d3-b707-dcc9eec2728a/node-exporter/0.log" Jul 07 17:10:14.948694 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.948664 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtq8q_052e5cc4-8625-46d3-b707-dcc9eec2728a/kube-rbac-proxy/0.log" Jul 07 17:10:14.976741 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:14.976713 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jtq8q_052e5cc4-8625-46d3-b707-dcc9eec2728a/init-textfile/0.log" Jul 07 17:10:15.185569 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.185535 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7glx6_69dc5200-1e97-473d-93a0-a02b6abe5654/kube-rbac-proxy-main/0.log" Jul 07 17:10:15.211434 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.211405 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7glx6_69dc5200-1e97-473d-93a0-a02b6abe5654/kube-rbac-proxy-self/0.log" Jul 07 17:10:15.248377 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.248340 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7glx6_69dc5200-1e97-473d-93a0-a02b6abe5654/openshift-state-metrics/0.log" Jul 07 17:10:15.618273 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.618245 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66dfb77967-cz7j7_05c7f4a6-4bf9-4fd8-beb5-24943180e7ea/telemeter-client/0.log" Jul 07 17:10:15.646092 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.645993 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66dfb77967-cz7j7_05c7f4a6-4bf9-4fd8-beb5-24943180e7ea/reload/0.log" Jul 07 17:10:15.672687 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:15.672661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-66dfb77967-cz7j7_05c7f4a6-4bf9-4fd8-beb5-24943180e7ea/kube-rbac-proxy/0.log" Jul 07 17:10:16.651020 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.650952 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nkp5f/must-gather-2m8jx" podStartSLOduration=11.908493579 podStartE2EDuration="12.650928606s" totalImagesPullingTime="742.435027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:10:06.244894809 +0000 UTC m=+2810.553734087" watchObservedRunningTime="2026-07-07 17:10:16.650928606 +0000 UTC m=+2820.959767885" Jul 07 17:10:16.651504 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.651201 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87"] Jul 07 17:10:16.658227 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.658198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.665320 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.665293 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87"] Jul 07 17:10:16.837143 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.837103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-sys\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.837394 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.837376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-lib-modules\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.837508 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.837493 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9b4\" (UniqueName: \"kubernetes.io/projected/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-kube-api-access-2q9b4\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.837685 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.837671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-proc\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.837891 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.837875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-podres\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.938887 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-proc\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.938887 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-podres\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939117 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-sys\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939117 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-lib-modules\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939117 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938987 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-podres\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939329 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.938990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9b4\" (UniqueName: \"kubernetes.io/projected/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-kube-api-access-2q9b4\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939329 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.939010 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-sys\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939329 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.939254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-proc\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.939329 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.939131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-lib-modules\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.950137 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.950076 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9b4\" (UniqueName: \"kubernetes.io/projected/52d9a54c-ce06-4b2a-a6e7-703c0ec481bd-kube-api-access-2q9b4\") pod \"perf-node-gather-daemonset-jff87\" (UID: \"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd\") " pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:16.971469 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:16.971436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:17.084048 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.084023 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-5qcwh_3af45c32-632e-4dbb-8f65-9a30f443b58f/networking-console-plugin/0.log" Jul 07 17:10:17.150636 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.150595 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87"] Jul 07 17:10:17.360190 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.360148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" event={"ID":"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd","Type":"ContainerStarted","Data":"150ce96d9a2a6f0f3295ef2aa346ecca4c845b94945a7c7f67a18073c3091e9d"} Jul 07 17:10:17.360382 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.360218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" event={"ID":"52d9a54c-ce06-4b2a-a6e7-703c0ec481bd","Type":"ContainerStarted","Data":"8e36164bb0017f661c81408b03ae344090cec98903d92d4235c47ade233a232b"} Jul 07 17:10:17.360450 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.360381 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:17.673145 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.673110 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/1.log" Jul 07 17:10:17.681091 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:17.680515 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-prqd8_3cb82cc1-986b-4f4a-bfab-ff9c2dcd836f/console-operator/2.log" Jul 07 17:10:18.143539 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:18.143495 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c58cdc598-2pdkw_724689cd-6362-4993-b489-5d07b366a4f4/console/0.log" Jul 07 17:10:18.182137 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:18.182113 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-7h5hw_819f7b33-44b8-4771-8329-e3216cc59113/download-server/0.log" Jul 07 17:10:18.775591 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:18.775560 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-rwpxg_f01eb24c-5045-42d7-9663-2e4724d5b361/volume-data-source-validator/0.log" Jul 07 17:10:19.900189 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:19.900162 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7htvg_7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4/dns/0.log" Jul 07 17:10:19.931608 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:19.931579 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7htvg_7310a4c3-ee87-4ac3-ad45-a30fda4fb6f4/kube-rbac-proxy/0.log" Jul 07 17:10:20.111671 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:20.111637 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5fq8_c9a61f94-1bf2-4834-b9d3-e90c846494c1/dns-node-resolver/0.log" Jul 07 17:10:20.797969 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:20.797938 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wnbfm_0e9f2128-48f5-4b83-816b-eba10e7a1bf2/node-ca/0.log" Jul 07 17:10:22.097165 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:22.097136 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-mmjx5_9827d53c-ce10-4438-b9d5-ccd350c49338/kube-auth-proxy/0.log" Jul 07 17:10:22.222977 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:22.222936 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-zb5zb_7292ff97-1198-4bce-8f35-2c4da06d79e2/payload-pre-processing/0.log" Jul 07 17:10:22.282267 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:22.282230 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-84j4f_960c0798-0406-4a62-b695-bc3a4fef2020/payload-processing/0.log" Jul 07 17:10:23.070897 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.070868 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vljgp_f07ac95b-22f0-41cc-8d29-c68702c78185/serve-healthcheck-canary/0.log" Jul 07 17:10:23.382233 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.382207 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" Jul 07 17:10:23.404580 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.404519 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nkp5f/perf-node-gather-daemonset-jff87" podStartSLOduration=7.404501116 podStartE2EDuration="7.404501116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:10:17.383323029 +0000 UTC m=+2821.692162298" watchObservedRunningTime="2026-07-07 17:10:23.404501116 +0000 UTC m=+2827.713340394" Jul 07 17:10:23.551408 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.551349 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-zmhwf_9b9be938-cda2-46fb-bab1-87b3b8584302/insights-operator/0.log" Jul 07 17:10:23.552045 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.552018 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-zmhwf_9b9be938-cda2-46fb-bab1-87b3b8584302/insights-operator/1.log" Jul 07 17:10:23.735788 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.735693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk524_e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df/kube-rbac-proxy/0.log" Jul 07 17:10:23.759332 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.759299 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk524_e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df/exporter/0.log" Jul 07 17:10:23.784052 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:23.784026 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mk524_e0c4b12a-6f4f-4cb6-ba31-5ea4c3a706df/extractor/0.log" Jul 07 17:10:26.585148 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:26.585111 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724045-b6hdd_229e8f96-74fc-439e-b82c-8d4c45079524/cleanup/0.log" Jul 07 17:10:26.627010 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:26.626927 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724060-gngmd_9efd8c50-80bd-4fc3-b91f-9379e4d54355/cleanup/0.log" Jul 07 17:10:26.784721 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:26.784674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56dc7d6c67-kcgvj_54193f19-612e-452b-8914-07912920c6b7/manager/0.log" Jul 07 17:10:27.273330 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.273249 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-cwsf9_ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766/manager/0.log" Jul 07 17:10:27.315880 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.315858 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fb6d99899-cwsf9_ad4474f3-0f1e-4e32-a2c7-3a12cc3bd766/copy-manifests/0.log" Jul 07 17:10:27.357879 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.357854 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-n97dk_1d973308-f677-4de8-9a4d-de98aeefd85d/postgres/0.log" Jul 07 17:10:27.363883 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.363863 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-mmjx5_9827d53c-ce10-4438-b9d5-ccd350c49338/kube-auth-proxy/0.log" Jul 07 17:10:27.465266 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.465243 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-zb5zb_7292ff97-1198-4bce-8f35-2c4da06d79e2/payload-pre-processing/0.log" Jul 07 17:10:27.511510 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:27.511483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-84j4f_960c0798-0406-4a62-b695-bc3a4fef2020/payload-processing/0.log" Jul 07 17:10:28.844920 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:28.844896 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-778798b69f-vxtmq_f4719801-3bbe-4bf3-bb73-1ff76c2d93ae/manager/0.log" Jul 07 17:10:28.957688 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:28.957661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfd697d7-mmjx5_9827d53c-ce10-4438-b9d5-ccd350c49338/kube-auth-proxy/0.log" Jul 07 17:10:29.070089 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:29.070058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-zb5zb_7292ff97-1198-4bce-8f35-2c4da06d79e2/payload-pre-processing/0.log" Jul 07 17:10:29.117998 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:29.117715 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-84j4f_960c0798-0406-4a62-b695-bc3a4fef2020/payload-processing/0.log" Jul 07 17:10:33.934359 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:33.934294 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4gs96_b705125e-6205-4969-94fa-92aa8e6dafd3/migrator/0.log" Jul 07 17:10:33.961460 ip-10-0-142-74 kubenswrapper[2569]: I0707 17:10:33.961425 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4gs96_b705125e-6205-4969-94fa-92aa8e6dafd3/graceful-termination/0.log"