Jul 07 04:22:05.700229 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 04:22:05.700239 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 04:22:05.700246 ip-10-0-142-139 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 04:22:05.700482 ip-10-0-142-139 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 04:22:15.432697 ip-10-0-142-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 04:22:15.432714 ip-10-0-142-139 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b02266dc62024510b173b3bccbf64984 -- Jul 07 04:24:00.745789 ip-10-0-142-139 systemd[1]: Starting Kubernetes Kubelet... Jul 07 04:24:02.999180 ip-10-0-142-139 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:24:02.999180 ip-10-0-142-139 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 04:24:02.999180 ip-10-0-142-139 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:24:02.999180 ip-10-0-142-139 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 04:24:02.999180 ip-10-0-142-139 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 04:24:02.999836 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:02.999272 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 04:24:03.005047 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005024 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.005047 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005045 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.005047 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005049 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005052 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005056 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005061 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005063 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005066 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005069 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005072 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005074 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005077 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005080 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005082 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005085 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005087 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005090 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005093 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005095 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005098 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005100 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005104 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.005131 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005107 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005110 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005112 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005115 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005118 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005120 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005123 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005125 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005128 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005131 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005134 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005136 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005142 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005146 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005149 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005152 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005156 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005158 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005161 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.005624 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005165 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005168 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005171 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005174 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005176 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005179 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005181 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005184 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005186 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005189 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005191 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005194 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005196 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005199 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005201 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005205 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005207 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005210 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005213 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005216 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.006120 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005218 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005221 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005223 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005226 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005229 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005231 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005234 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005237 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005239 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005242 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005245 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005248 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005250 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005253 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005256 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005259 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005261 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005264 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005266 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.006592 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005269 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005273 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005276 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005279 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005281 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005284 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005286 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005289 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005293 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005295 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005298 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005303 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005306 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005308 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005311 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005314 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005317 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005320 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.005322 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006056 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.007048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006063 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006066 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006068 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006071 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006075 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006078 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006080 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006083 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006086 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006088 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006091 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006093 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006096 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006099 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006101 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006104 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006106 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006109 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006112 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006114 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.007554 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006118 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006121 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006124 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006127 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006130 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006133 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006136 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006139 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006141 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006144 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006146 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006149 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006151 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006156 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006160 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006163 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006167 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006171 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006174 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.008081 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006177 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006180 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006183 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006185 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006188 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006191 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006194 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006196 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006199 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006202 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006205 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006208 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006211 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006213 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006219 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006222 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006225 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006228 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006232 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.008532 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006234 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006237 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006240 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006243 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006246 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006248 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006251 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006253 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006256 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006258 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006261 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006263 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006266 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006269 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006271 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006275 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006280 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006283 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006286 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006289 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.008986 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006291 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006294 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006296 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006299 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006302 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006305 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006307 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006309 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006312 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006315 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006318 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006321 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006323 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006326 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006328 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006331 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006334 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006337 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006339 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.009460 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006342 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006754 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006759 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006762 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006766 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006769 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006772 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006775 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006778 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006781 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006784 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006786 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006789 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006792 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006794 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006797 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006799 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006804 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006807 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006811 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.009902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006814 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006817 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006819 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006822 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006825 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006827 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006830 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006832 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006835 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006838 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006840 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006843 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006845 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006848 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006850 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006853 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006855 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006858 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006863 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006866 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.010406 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006869 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006871 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006874 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006877 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006879 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006882 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006884 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006887 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006889 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006895 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006897 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006901 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006904 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006908 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006912 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006915 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006917 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006920 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006922 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006925 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.010879 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006927 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006930 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006932 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006935 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006937 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006940 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006942 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006945 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006949 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006952 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006955 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006958 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006961 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006964 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006967 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006970 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006973 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006976 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006981 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.011358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006984 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006987 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006991 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006994 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.006998 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007015 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007018 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007021 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007023 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007026 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007029 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007032 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007035 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007038 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007041 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007043 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007046 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007049 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007052 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007055 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.011793 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007058 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007127 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007134 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007141 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007146 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007151 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007154 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007159 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007164 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007167 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007170 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007173 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007177 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007180 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007183 2560 flags.go:64] FLAG: --cgroup-root="" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007186 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007191 2560 flags.go:64] FLAG: --client-ca-file="" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007194 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007197 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007201 2560 flags.go:64] FLAG: --cluster-domain="" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007203 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007206 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007210 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007214 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 04:24:03.012289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007217 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007220 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007223 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007227 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007230 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007233 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007235 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007239 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007242 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007246 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007249 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007251 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007254 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007257 2560 flags.go:64] FLAG: --enable-server="true" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007260 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007265 2560 flags.go:64] FLAG: --event-burst="100" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007268 2560 flags.go:64] FLAG: --event-qps="50" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007271 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007274 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007276 2560 flags.go:64] FLAG: --eviction-hard="" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007280 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007283 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007286 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007289 2560 flags.go:64] FLAG: --eviction-soft="" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007294 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 04:24:03.012860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007297 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007300 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007303 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007306 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007309 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007312 2560 flags.go:64] FLAG: --feature-gates="" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007316 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007319 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007322 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007325 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007328 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007331 2560 flags.go:64] FLAG: --help="false" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007334 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007337 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007340 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007343 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007346 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007349 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007352 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007355 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007358 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007361 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007364 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007367 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 04:24:03.013534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007370 2560 flags.go:64] FLAG: --kube-reserved="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007373 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007375 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007378 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007381 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007384 2560 flags.go:64] FLAG: --lock-file="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007387 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007391 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007394 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007400 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007403 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007405 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007408 2560 flags.go:64] FLAG: --logging-format="text" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007411 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007414 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007418 2560 flags.go:64] FLAG: --manifest-url="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007421 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007425 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007428 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007432 2560 flags.go:64] FLAG: --max-pods="110" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007435 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007438 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007441 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007444 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007447 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 04:24:03.014107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007450 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007453 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007460 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007463 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007466 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007470 2560 flags.go:64] FLAG: --pod-cidr="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007473 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007479 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007482 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007485 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007488 2560 flags.go:64] FLAG: --port="10250" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007491 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007494 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06e2302301e2ae357" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007497 2560 flags.go:64] FLAG: --qos-reserved="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007501 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007504 2560 flags.go:64] FLAG: --register-node="true" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007507 2560 flags.go:64] FLAG: --register-with-taints="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007511 2560 flags.go:64] FLAG: --registry-burst="10" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007513 2560 flags.go:64] FLAG: --registry-qps="5" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007516 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007519 2560 flags.go:64] FLAG: --reserved-memory="" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007522 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007525 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007528 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007531 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 04:24:03.014710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007534 2560 flags.go:64] FLAG: --runonce="false" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007537 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007540 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007543 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007546 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007549 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007552 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007555 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007558 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007562 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007565 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007568 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007570 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007573 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007577 2560 flags.go:64] FLAG: --system-cgroups="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007580 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007585 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007588 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007591 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007595 2560 flags.go:64] FLAG: --tls-min-version="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007599 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007603 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007605 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007608 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007611 2560 flags.go:64] FLAG: --v="2" Jul 07 04:24:03.015319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007615 2560 flags.go:64] FLAG: --version="false" Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007619 2560 flags.go:64] FLAG: --vmodule="" Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007623 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.007626 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007779 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007784 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007787 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007790 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007792 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007795 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007797 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007800 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007802 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007805 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007808 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007810 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007813 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007815 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007818 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007820 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007823 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.015902 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007825 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007828 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007831 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007834 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007837 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007839 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007843 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007847 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007850 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007852 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007855 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007858 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007860 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007863 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007865 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007868 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007871 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007873 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007876 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007879 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.016424 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007881 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007884 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007886 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007890 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007894 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007897 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007900 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007902 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007905 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007907 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007910 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007913 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007915 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007929 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007934 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007937 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007939 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007942 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007946 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.016933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007949 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007952 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007954 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007957 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007959 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007962 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007964 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007967 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007970 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007972 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007975 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007977 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007980 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007982 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007985 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007987 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007990 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007992 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007995 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.017421 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.007997 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008012 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008015 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008018 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008020 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008023 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008025 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008028 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008032 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008036 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008040 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008044 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008048 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008051 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008054 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008056 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008059 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008062 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008064 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.017933 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008067 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008070 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008073 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008075 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.008078 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.008991 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017142 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017161 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017181 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017187 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017265 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017270 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017274 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017278 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017281 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.018411 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017284 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017287 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017289 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017292 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017295 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017298 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017300 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017304 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017307 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017309 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017312 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017314 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017317 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017320 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017323 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017326 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017328 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017331 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017334 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017336 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.018777 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017339 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017341 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017344 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017346 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017350 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017354 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017356 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017360 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017364 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017367 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017370 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017373 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017376 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017379 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017382 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017384 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017388 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017392 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017396 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.019270 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017398 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017402 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017405 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017407 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017410 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017413 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017415 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017418 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017420 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017423 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017426 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017429 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017432 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017435 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017437 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017440 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017442 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017446 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017448 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017451 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.019718 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017454 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017456 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017459 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017464 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017467 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017471 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017474 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017477 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017479 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017482 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017485 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017488 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017491 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017494 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017497 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017500 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017502 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017505 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017509 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.020430 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017511 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017514 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017516 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017519 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017522 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017524 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017527 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017529 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017532 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017535 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017538 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017540 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017543 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017546 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017548 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.021150 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017551 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017556 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017673 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017678 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017681 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017684 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017687 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017690 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017692 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017695 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017697 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017700 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017702 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017705 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017708 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 04:24:03.021512 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017711 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017714 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017716 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017719 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017721 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017724 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017727 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017729 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017732 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017734 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017736 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017739 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017742 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017744 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017747 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017750 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017753 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017755 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017758 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017761 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 04:24:03.021897 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017763 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017766 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017768 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017771 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017773 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017776 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017779 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017781 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017784 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017786 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017788 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017792 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017794 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017797 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017799 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017802 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017804 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017807 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017809 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017812 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 04:24:03.022385 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017815 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017817 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017820 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017822 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017828 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017831 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017833 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017836 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017839 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017842 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017844 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017847 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017849 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017852 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017854 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017856 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017859 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017861 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017865 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 04:24:03.023023 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017868 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017871 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017874 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017877 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017880 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017883 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017887 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017890 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017893 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017896 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017899 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017902 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017904 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017908 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017912 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017915 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017917 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017921 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017924 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 04:24:03.023632 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017926 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017929 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017931 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017934 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017937 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017940 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017942 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:03.017946 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.017951 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.018093 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 04:24:03.024161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.023782 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 04:24:03.025206 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.025193 2560 server.go:1019] "Starting client certificate rotation" Jul 07 04:24:03.025309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.025291 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 04:24:03.025354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.025319 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 04:24:03.028417 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.028403 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 04:24:03.029629 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.029610 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 04:24:03.037445 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.037426 2560 log.go:25] "Validated CRI v1 runtime API" Jul 07 04:24:03.040427 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.040408 2560 log.go:25] "Validated CRI v1 image API" Jul 07 04:24:03.041271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.041256 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 04:24:03.043938 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.043917 2560 fs.go:135] Filesystem UUIDs: map[40b04be3-3ad6-4f5c-afa0-7c2b6d48c424:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7aea8f0-8511-4b02-9568-ad0f663f98cd:/dev/nvme0n1p3] Jul 07 04:24:03.044024 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.043936 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 04:24:03.050022 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.049890 2560 manager.go:216] Machine: {Timestamp:2026-07-07 04:24:03.048594427 +0000 UTC m=+0.716012610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097113 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ace5d7f5da0d383898f5ca5126f2d SystemUUID:ec2ace5d-7f5d-a0d3-8389-8f5ca5126f2d BootID:b02266dc-6202-4510-b173-b3bccbf64984 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:5a:b2:c0:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:5a:b2:c0:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:4b:74:af:c3:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 04:24:03.050022 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.050015 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 04:24:03.050150 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.050100 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 04:24:03.053015 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.052976 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 04:24:03.053512 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.053115 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 04:24:03.055191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.055172 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 04:24:03.055277 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.055195 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 04:24:03.055277 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.055207 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 04:24:03.055997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.055975 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 04:24:03.057766 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.057753 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 04:24:03.058698 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.058682 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 04:24:03.060715 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060703 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 07 04:24:03.060827 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060817 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 04:24:03.060855 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060849 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 07 04:24:03.060882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060861 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 04:24:03.060882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060872 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 04:24:03.060882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060881 2560 kubelet.go:398] "Adding apiserver pod source" Jul 07 04:24:03.060982 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.060890 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 04:24:03.061271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.061260 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 04:24:03.061325 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.061278 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 04:24:03.063918 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.063896 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 04:24:03.065915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.065877 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 04:24:03.065915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.065907 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 04:24:03.066851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066839 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066856 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066863 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066870 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066879 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066885 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 04:24:03.066892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066894 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 04:24:03.067080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066900 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 04:24:03.067080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066907 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 04:24:03.067080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066912 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 04:24:03.067080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066921 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 04:24:03.067080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.066926 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 04:24:03.067656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.067646 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 04:24:03.067688 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.067656 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 04:24:03.070485 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.070460 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 04:24:03.070485 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.070482 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 04:24:03.070583 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.070469 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 04:24:03.071040 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.071028 2560 server.go:1268] "Started kubelet" Jul 07 04:24:03.071140 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.071097 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 04:24:03.071202 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.071167 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 04:24:03.071246 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.071115 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 04:24:03.071946 ip-10-0-142-139 systemd[1]: Started Kubernetes Kubelet. Jul 07 04:24:03.072762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.072711 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 04:24:03.073900 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.073884 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 07 04:24:03.076911 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.076891 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 04:24:03.078643 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.078629 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 04:24:03.078730 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.078708 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 04:24:03.079271 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.078821 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.079507 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079487 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 04:24:03.079507 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079507 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 04:24:03.079639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079487 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 04:24:03.079639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079589 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 07 04:24:03.079639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079591 2560 factory.go:55] Registering systemd factory Jul 07 04:24:03.079639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079606 2560 factory.go:223] Registration of the systemd container factory successfully Jul 07 04:24:03.079829 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079596 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 07 04:24:03.079881 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079851 2560 factory.go:153] Registering CRI-O factory Jul 07 04:24:03.079881 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079866 2560 factory.go:223] Registration of the crio container factory successfully Jul 07 04:24:03.079969 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079912 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 04:24:03.079969 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079932 2560 factory.go:103] Registering Raw factory Jul 07 04:24:03.079969 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.079947 2560 manager.go:1196] Started watching for new ooms in manager Jul 07 04:24:03.080401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.080366 2560 manager.go:349] Starting recovery of all containers Jul 07 04:24:03.080986 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.079902 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-139.ec2.internal.18bfe740d34d37c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-139.ec2.internal,UID:ip-10-0-142-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-139.ec2.internal,},FirstTimestamp:2026-07-07 04:24:03.07098208 +0000 UTC m=+0.738400263,LastTimestamp:2026-07-07 04:24:03.07098208 +0000 UTC m=+0.738400263,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-139.ec2.internal,}" Jul 07 04:24:03.082299 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.082270 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 04:24:03.082418 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.082396 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 04:24:03.086226 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.086201 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fclms" Jul 07 04:24:03.092130 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.091965 2560 manager.go:354] Recovery completed Jul 07 04:24:03.092799 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.092781 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fclms" Jul 07 04:24:03.096510 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.096493 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.098278 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098262 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.098353 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098293 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.098353 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098303 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.098843 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098827 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 04:24:03.098843 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098842 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 04:24:03.098946 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098828 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.098946 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.098862 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 07 04:24:03.099500 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099485 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.099568 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099516 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.099568 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099530 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.099861 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099849 2560 policy_none.go:49] "None policy: Start" Jul 07 04:24:03.099901 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099866 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 04:24:03.099901 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.099876 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 04:24:03.100599 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.100527 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-139.ec2.internal.18bfe740d4edbca7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-139.ec2.internal,UID:ip-10-0-142-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-139.ec2.internal,},FirstTimestamp:2026-07-07 04:24:03.098279079 +0000 UTC m=+0.765697262,LastTimestamp:2026-07-07 04:24:03.098279079 +0000 UTC m=+0.765697262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-139.ec2.internal,}" Jul 07 04:24:03.100797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.100783 2560 policy_none.go:47] "Start" Jul 07 04:24:03.137723 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.137562 2560 manager.go:337] "Starting Device Plugin manager" Jul 07 04:24:03.137861 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.137762 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 04:24:03.137861 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.137776 2560 server.go:87] "Starting device plugin registration server" Jul 07 04:24:03.138082 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.138069 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 04:24:03.138145 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.138085 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 04:24:03.138236 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.138216 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 04:24:03.138323 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.138308 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 04:24:03.138323 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.138322 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 04:24:03.138840 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.138818 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 04:24:03.138926 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.138865 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.228448 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.228417 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 04:24:03.229706 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.229688 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 04:24:03.229778 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.229735 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 04:24:03.229778 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.229760 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 04:24:03.229856 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.229793 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 04:24:03.232024 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.231990 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:24:03.238487 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.238472 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.239315 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.239301 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.239385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.239329 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.239385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.239339 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.239385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.239366 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.247634 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.247617 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.247698 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.247642 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-139.ec2.internal\": node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.266735 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.266708 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.330819 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.330730 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"] Jul 07 04:24:03.330914 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.330823 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.332432 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.332415 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.332501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.332446 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.332501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.332455 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.333676 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.333661 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.333797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.333783 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.333848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.333813 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.334474 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334461 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.334548 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334485 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.334548 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334506 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.334955 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334941 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.335014 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334965 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.335014 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.334975 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.336014 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.335989 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.336062 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.336032 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 04:24:03.336720 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.336707 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 07 04:24:03.336785 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.336740 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 04:24:03.336785 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.336750 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 07 04:24:03.357929 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.357904 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.362625 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.362604 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.366895 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.366875 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.381119 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.381079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.381262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.381152 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.381262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.381179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebfeb482643ae23298dcd20cee13bf5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"7ebfeb482643ae23298dcd20cee13bf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.467718 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.467687 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.482101 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.482255 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebfeb482643ae23298dcd20cee13bf5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"7ebfeb482643ae23298dcd20cee13bf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.482255 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.482255 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482189 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.482365 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482254 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/311c3c33db1bb18e1e66fdc4b1c32e51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"311c3c33db1bb18e1e66fdc4b1c32e51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.482365 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.482292 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebfeb482643ae23298dcd20cee13bf5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"7ebfeb482643ae23298dcd20cee13bf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.568573 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.568529 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.661137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.661061 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.665317 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:03.664861 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:03.669036 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.669017 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.769659 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.769616 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.870144 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.870113 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:03.970795 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:03.970715 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:04.025222 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.025187 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 04:24:04.025734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.025351 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 04:24:04.070865 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:04.070830 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 07 04:24:04.095984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.095944 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 04:19:03 +0000 UTC" deadline="2028-01-10 08:50:12.383316691 +0000 UTC" Jul 07 04:24:04.095984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.095980 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13252h26m8.287340793s" Jul 07 04:24:04.103866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.103846 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:24:04.179676 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.179648 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 07 04:24:04.193131 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.193103 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 04:24:04.194972 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.194953 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 07 04:24:04.204190 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:04.204150 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebfeb482643ae23298dcd20cee13bf5.slice/crio-6e34eef8d70649cbb1edc7d9fe647f38ca177fb3ff15f4aa0a963344ce9efc90 WatchSource:0}: Error finding container 6e34eef8d70649cbb1edc7d9fe647f38ca177fb3ff15f4aa0a963344ce9efc90: Status 404 returned error can't find the container with id 6e34eef8d70649cbb1edc7d9fe647f38ca177fb3ff15f4aa0a963344ce9efc90 Jul 07 04:24:04.204809 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:04.204779 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311c3c33db1bb18e1e66fdc4b1c32e51.slice/crio-af90d323e4695333325a33ab8571199a334309c4ca822c2266879a8a117be061 WatchSource:0}: Error finding container af90d323e4695333325a33ab8571199a334309c4ca822c2266879a8a117be061: Status 404 returned error can't find the container with id af90d323e4695333325a33ab8571199a334309c4ca822c2266879a8a117be061 Jul 07 04:24:04.205635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.205613 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 04:24:04.209744 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.209726 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:24:04.233705 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.233593 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"7ebfeb482643ae23298dcd20cee13bf5","Type":"ContainerStarted","Data":"6e34eef8d70649cbb1edc7d9fe647f38ca177fb3ff15f4aa0a963344ce9efc90"} Jul 07 04:24:04.234821 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.234801 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"311c3c33db1bb18e1e66fdc4b1c32e51","Type":"ContainerStarted","Data":"af90d323e4695333325a33ab8571199a334309c4ca822c2266879a8a117be061"} Jul 07 04:24:04.404628 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.404607 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:24:04.468045 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:04.467989 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:24:05.061614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.061578 2560 apiserver.go:52] "Watching apiserver" Jul 07 04:24:05.068921 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.068889 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 04:24:05.072493 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.072464 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 04:24:05.073865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.073839 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-57qdr","kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal","openshift-cluster-node-tuning-operator/tuned-98hh5","openshift-image-registry/node-ca-9ttvr","openshift-multus/multus-v7d99","openshift-multus/network-metrics-daemon-mdqmk","openshift-ovn-kubernetes/ovnkube-node-vdn94","kube-system/global-pull-secret-syncer-tx8w2","kube-system/konnectivity-agent-gxjfc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx","openshift-dns/node-resolver-bnslv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-7xfmc","openshift-network-diagnostics/network-check-target-zd488"] Jul 07 04:24:05.075390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.075368 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.076485 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.076463 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.078294 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.078252 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.078851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.078699 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 04:24:05.078851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.078757 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fjm6l\"" Jul 07 04:24:05.078851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.078795 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 04:24:05.078851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.078817 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.079350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.079333 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.079436 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.079420 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 04:24:05.079637 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.079617 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080243 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080437 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080473 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080678 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080719 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080808 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080946 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.081106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.080958 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7lpgd\"" Jul 07 04:24:05.082023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.081839 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.082023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.081905 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.082023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.081958 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.082023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.081908 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9m5k7\"" Jul 07 04:24:05.082327 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.082174 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.082581 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.082564 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4cpvg\"" Jul 07 04:24:05.083510 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.083082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.083510 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.083177 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:05.083510 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.083328 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 04:24:05.085532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.084282 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tt8j8\"" Jul 07 04:24:05.085532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.084474 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.085532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.084675 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 04:24:05.085532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.085080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.085758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.085531 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.086746 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.086727 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:05.086857 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.086831 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:05.088321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.088302 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.089386 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.089793 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.089845 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090278 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x9lg2\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090337 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-69shv\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090384 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090595 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-tuned\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090626 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-log-socket\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-run\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090696 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-lib-modules\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ababb864-9ac2-497c-b428-cff38e27581a-serviceca\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090784 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-os-release\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090791 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090844 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-multus\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090891 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090915 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-etc-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.090949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-modprobe-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-var-lib-kubelet\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.091444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091192 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-tmp\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091218 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysconfig\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091325 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-cnibin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovn-node-metrics-cert\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091416 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-system-cni-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091444 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-system-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091471 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-socket-dir-parent\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-k8s-cni-cncf-io\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091563 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-kubelet\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091659 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091672 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-cnibin\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091706 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091842 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-bin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-systemd-units\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.092447 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ababb864-9ac2-497c-b428-cff38e27581a-host\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-var-lib-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.091989 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnsjj\"" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092054 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjmx\" (UniqueName: \"kubernetes.io/projected/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-kube-api-access-6rjmx\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092107 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-kubernetes\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-os-release\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092285 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-cni-binary-copy\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092323 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-multus-certs\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092419 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-node-log\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092457 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-sys\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-host\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-daemon-config\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-bin\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99sq\" (UniqueName: \"kubernetes.io/projected/b2a084a3-547a-4578-bd94-675420fca735-kube-api-access-q99sq\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092679 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5vlf\" (UniqueName: \"kubernetes.io/projected/ababb864-9ac2-497c-b428-cff38e27581a-kube-api-access-s5vlf\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092720 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-conf-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.093309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.092749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-systemd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093982 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.093460 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-netd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.093982 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.093542 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.093982 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.093544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.094156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094022 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-systemd\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.094156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094051 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-etc-kubernetes\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.094156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-kubelet\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-netns\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094134 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-script-lib\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094396 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094159 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszfm\" (UniqueName: \"kubernetes.io/projected/acbacb48-194a-4e8d-acff-2310e35c7b03-kube-api-access-zszfm\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.094396 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094192 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-netns\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.094396 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094221 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-slash\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094396 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-config\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094396 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.094731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094292 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-ovn\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094643 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-env-overrides\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.094731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-conf\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.094731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-hostroot\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.094911 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmhr\" (UniqueName: \"kubernetes.io/projected/c406620d-9bcd-476c-90b4-ec9e51b25c45-kube-api-access-jhmhr\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.094985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.094966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.095508 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.095349 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 04:24:05.096230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.096213 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.096320 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.096275 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:05.099635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.099616 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 04:24:05.099852 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.099835 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jmdfk\"" Jul 07 04:24:05.099918 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.099889 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 04:24:05.126257 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.126225 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwjp9" Jul 07 04:24:05.135181 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.135153 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwjp9" Jul 07 04:24:05.181096 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.181063 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 04:24:05.196141 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196106 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-ovn\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196141 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-env-overrides\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgshv\" (UniqueName: \"kubernetes.io/projected/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-kube-api-access-cgshv\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-conf\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196231 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-hostroot\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196231 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-ovn\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196250 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmhr\" (UniqueName: \"kubernetes.io/projected/c406620d-9bcd-476c-90b4-ec9e51b25c45-kube-api-access-jhmhr\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196293 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-hostroot\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.196381 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196372 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-tuned\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196398 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-log-socket\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196402 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-conf\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196402 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196438 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-log-socket\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/534725e6-d911-4e9d-bee9-a90ceb824a86-agent-certs\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-run\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196517 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-lib-modules\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ababb864-9ac2-497c-b428-cff38e27581a-serviceca\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196590 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-os-release\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196680 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-lib-modules\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196701 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-run\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196709 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-os-release\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196694 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196738 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-env-overrides\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfgn\" (UniqueName: \"kubernetes.io/projected/0134d13a-7be7-4059-ad00-f6cb9898f4e5-kube-api-access-dlfgn\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.196858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-kubelet-config\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-multus\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-etc-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196857 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-multus\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzv27\" (UniqueName: \"kubernetes.io/projected/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-kube-api-access-hzv27\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196903 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-etc-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-device-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196944 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196966 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-modprobe-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.196989 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-var-lib-kubelet\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197024 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ababb864-9ac2-497c-b428-cff38e27581a-serviceca\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-tmp\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197061 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197097 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-var-lib-kubelet\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197114 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-modprobe-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197109 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-registration-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.197647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197142 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-ovn-kubernetes\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysconfig\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197181 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-cnibin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197207 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovn-node-metrics-cert\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197258 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysconfig\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197258 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-cnibin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197315 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-system-cni-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-system-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-socket-dir-parent\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-k8s-cni-cncf-io\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197434 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-kubelet\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197438 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-system-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197388 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-system-cni-dir\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197458 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-cnibin\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197498 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.198343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-bin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197563 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-systemd-units\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197578 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-cni-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-sys-fs\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197593 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-k8s-cni-cncf-io\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197614 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ababb864-9ac2-497c-b428-cff38e27581a-host\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197627 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-socket-dir-parent\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197636 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-var-lib-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197563 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-kubelet\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197669 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjmx\" (UniqueName: \"kubernetes.io/projected/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-kube-api-access-6rjmx\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197673 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-cnibin\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197684 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-var-lib-openvswitch\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197716 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-systemd-units\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-dbus\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197730 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ababb864-9ac2-497c-b428-cff38e27581a-host\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/534725e6-d911-4e9d-bee9-a90ceb824a86-konnectivity-ca\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.199216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197770 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-kubernetes\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-os-release\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197835 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-cni-binary-copy\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197857 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-multus-certs\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197905 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-kubernetes\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197910 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-cni-binary-copy\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197930 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-var-lib-cni-bin\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197958 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2a084a3-547a-4578-bd94-675420fca735-os-release\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-node-log\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.197975 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-multus-certs\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198035 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-tmp-dir\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198063 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0134d13a-7be7-4059-ad00-f6cb9898f4e5-iptables-alerter-script\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198080 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-sys\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-host\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198122 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-node-log\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-daemon-config\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198182 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-sys\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.199932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198213 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-bin\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198253 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-socket-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198287 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198313 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99sq\" (UniqueName: \"kubernetes.io/projected/b2a084a3-547a-4578-bd94-675420fca735-kube-api-access-q99sq\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5vlf\" (UniqueName: \"kubernetes.io/projected/ababb864-9ac2-497c-b428-cff38e27581a-kube-api-access-s5vlf\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-conf-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-systemd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-netd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj4q\" (UniqueName: \"kubernetes.io/projected/be24b6de-a673-43bb-9b7b-55c29137a1f6-kube-api-access-sfj4q\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198480 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-cni-binary-copy\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198497 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-systemd\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198534 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-conf-dir\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-etc-kubernetes\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-kubelet\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198648 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-run-systemd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-netns\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.200729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198664 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c406620d-9bcd-476c-90b4-ec9e51b25c45-multus-daemon-config\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-script-lib\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198712 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198716 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-etc-kubernetes\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198717 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-systemd\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198758 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-bin\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198766 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-kubelet\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198619 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-host\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198790 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-cni-netd\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.198818 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-run-netns\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199162 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2a084a3-547a-4578-bd94-675420fca735-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszfm\" (UniqueName: \"kubernetes.io/projected/acbacb48-194a-4e8d-acff-2310e35c7b03-kube-api-access-zszfm\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-netns\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-slash\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-config\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0134d13a-7be7-4059-ad00-f6cb9898f4e5-host-slash\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199398 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-hosts-file\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.201538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199431 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199521 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c406620d-9bcd-476c-90b4-ec9e51b25c45-host-run-netns\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199561 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-host-slash\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199586 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-sysctl-d\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.199965 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-config\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.200067 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovnkube-script-lib\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.200958 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-tmp\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.200980 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acbacb48-194a-4e8d-acff-2310e35c7b03-etc-tuned\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.202337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.201073 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-ovn-node-metrics-cert\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.207772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.207740 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmhr\" (UniqueName: \"kubernetes.io/projected/c406620d-9bcd-476c-90b4-ec9e51b25c45-kube-api-access-jhmhr\") pod \"multus-v7d99\" (UID: \"c406620d-9bcd-476c-90b4-ec9e51b25c45\") " pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.207772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.207759 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjmx\" (UniqueName: \"kubernetes.io/projected/8f94c1df-25d3-4528-9f2b-d7b3d3dce402-kube-api-access-6rjmx\") pod \"ovnkube-node-vdn94\" (UID: \"8f94c1df-25d3-4528-9f2b-d7b3d3dce402\") " pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.208980 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.208950 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99sq\" (UniqueName: \"kubernetes.io/projected/b2a084a3-547a-4578-bd94-675420fca735-kube-api-access-q99sq\") pod \"multus-additional-cni-plugins-7xfmc\" (UID: \"b2a084a3-547a-4578-bd94-675420fca735\") " pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.209481 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.209335 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5vlf\" (UniqueName: \"kubernetes.io/projected/ababb864-9ac2-497c-b428-cff38e27581a-kube-api-access-s5vlf\") pod \"node-ca-9ttvr\" (UID: \"ababb864-9ac2-497c-b428-cff38e27581a\") " pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.209581 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.209566 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszfm\" (UniqueName: \"kubernetes.io/projected/acbacb48-194a-4e8d-acff-2310e35c7b03-kube-api-access-zszfm\") pod \"tuned-98hh5\" (UID: \"acbacb48-194a-4e8d-acff-2310e35c7b03\") " pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.299880 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299846 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.299880 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299883 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/534725e6-d911-4e9d-bee9-a90ceb824a86-agent-certs\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfgn\" (UniqueName: \"kubernetes.io/projected/0134d13a-7be7-4059-ad00-f6cb9898f4e5-kube-api-access-dlfgn\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-kubelet-config\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.299981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.300034 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzv27\" (UniqueName: \"kubernetes.io/projected/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-kube-api-access-hzv27\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-device-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300079 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-kubelet-config\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.300112 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.80008678 +0000 UTC m=+3.467504969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300132 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-device-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-registration-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300195 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-registration-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-sys-fs\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300320 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-sys-fs\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.300323 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300342 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-dbus\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.300375 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.800359239 +0000 UTC m=+3.467777415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300391 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/534725e6-d911-4e9d-bee9-a90ceb824a86-konnectivity-ca\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300412 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-dbus\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-tmp-dir\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300541 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0134d13a-7be7-4059-ad00-f6cb9898f4e5-iptables-alerter-script\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-socket-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj4q\" (UniqueName: \"kubernetes.io/projected/be24b6de-a673-43bb-9b7b-55c29137a1f6-kube-api-access-sfj4q\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300676 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-tmp-dir\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300686 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0134d13a-7be7-4059-ad00-f6cb9898f4e5-host-slash\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.300714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300717 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-hosts-file\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300721 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0134d13a-7be7-4059-ad00-f6cb9898f4e5-host-slash\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgshv\" (UniqueName: \"kubernetes.io/projected/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-kube-api-access-cgshv\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300773 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-hosts-file\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300866 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-socket-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.300919 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.301018 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/534725e6-d911-4e9d-bee9-a90ceb824a86-konnectivity-ca\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.301048 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be24b6de-a673-43bb-9b7b-55c29137a1f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.301416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.301114 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0134d13a-7be7-4059-ad00-f6cb9898f4e5-iptables-alerter-script\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.303031 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.302992 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/534725e6-d911-4e9d-bee9-a90ceb824a86-agent-certs\") pod \"konnectivity-agent-gxjfc\" (UID: \"534725e6-d911-4e9d-bee9-a90ceb824a86\") " pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.309273 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.309247 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:05.309273 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.309275 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:05.309450 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.309291 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:05.309450 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.309364 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:05.809347631 +0000 UTC m=+3.476765823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:05.310878 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.310849 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfgn\" (UniqueName: \"kubernetes.io/projected/0134d13a-7be7-4059-ad00-f6cb9898f4e5-kube-api-access-dlfgn\") pod \"iptables-alerter-57qdr\" (UID: \"0134d13a-7be7-4059-ad00-f6cb9898f4e5\") " pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.311609 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.311587 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzv27\" (UniqueName: \"kubernetes.io/projected/cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f-kube-api-access-hzv27\") pod \"node-resolver-bnslv\" (UID: \"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f\") " pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.314792 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.314731 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgshv\" (UniqueName: \"kubernetes.io/projected/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-kube-api-access-cgshv\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.315238 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.315217 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj4q\" (UniqueName: \"kubernetes.io/projected/be24b6de-a673-43bb-9b7b-55c29137a1f6-kube-api-access-sfj4q\") pod \"aws-ebs-csi-driver-node-g8kwx\" (UID: \"be24b6de-a673-43bb-9b7b-55c29137a1f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.389672 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.389634 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:05.398487 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.398456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" Jul 07 04:24:05.413269 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.413237 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-98hh5" Jul 07 04:24:05.417710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.417689 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ttvr" Jul 07 04:24:05.426260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.426239 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v7d99" Jul 07 04:24:05.433776 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.433752 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:05.443354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.443328 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57qdr" Jul 07 04:24:05.451441 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.451418 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" Jul 07 04:24:05.457687 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.457661 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bnslv" Jul 07 04:24:05.803171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.803080 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:05.803171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.803121 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:05.803368 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.803250 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.803368 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.803307 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:06.803287843 +0000 UTC m=+4.470706020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:05.803368 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.803251 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.803368 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.803346 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:06.80333689 +0000 UTC m=+4.470755075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:05.904041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:05.903993 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:05.904245 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.904173 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:05.904245 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.904205 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:05.904245 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.904217 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:05.904410 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:05.904279 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:06.904261339 +0000 UTC m=+4.571679513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:06.080591 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.080565 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a084a3_547a_4578_bd94_675420fca735.slice/crio-00a5621ae40a2d71f76b23a4f0a13b870f54e32a19c08eb742db9be132bebee9 WatchSource:0}: Error finding container 00a5621ae40a2d71f76b23a4f0a13b870f54e32a19c08eb742db9be132bebee9: Status 404 returned error can't find the container with id 00a5621ae40a2d71f76b23a4f0a13b870f54e32a19c08eb742db9be132bebee9 Jul 07 04:24:06.081667 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.081642 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podababb864_9ac2_497c_b428_cff38e27581a.slice/crio-a02cde89fa94ed005571dc5796e2447378385ab82eebd8dab72a28dd7f0be057 WatchSource:0}: Error finding container a02cde89fa94ed005571dc5796e2447378385ab82eebd8dab72a28dd7f0be057: Status 404 returned error can't find the container with id a02cde89fa94ed005571dc5796e2447378385ab82eebd8dab72a28dd7f0be057 Jul 07 04:24:06.082312 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.082280 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534725e6_d911_4e9d_bee9_a90ceb824a86.slice/crio-25f5e133de524797a610205ac3f4765aa5e5b659fb782a648548d554a39f63d4 WatchSource:0}: Error finding container 25f5e133de524797a610205ac3f4765aa5e5b659fb782a648548d554a39f63d4: Status 404 returned error can't find the container with id 25f5e133de524797a610205ac3f4765aa5e5b659fb782a648548d554a39f63d4 Jul 07 04:24:06.083238 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.083219 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbacb48_194a_4e8d_acff_2310e35c7b03.slice/crio-8f9c705a278ff3f93fbba57381fdea629de6663c0662d4bb1ac1de50554f59a6 WatchSource:0}: Error finding container 8f9c705a278ff3f93fbba57381fdea629de6663c0662d4bb1ac1de50554f59a6: Status 404 returned error can't find the container with id 8f9c705a278ff3f93fbba57381fdea629de6663c0662d4bb1ac1de50554f59a6 Jul 07 04:24:06.087504 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.087484 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe24b6de_a673_43bb_9b7b_55c29137a1f6.slice/crio-edc15899563df7f8eb461da6d40fde9bcdc230af4f8a60fa3d2193be83c7661a WatchSource:0}: Error finding container edc15899563df7f8eb461da6d40fde9bcdc230af4f8a60fa3d2193be83c7661a: Status 404 returned error can't find the container with id edc15899563df7f8eb461da6d40fde9bcdc230af4f8a60fa3d2193be83c7661a Jul 07 04:24:06.088416 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.088396 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc406620d_9bcd_476c_90b4_ec9e51b25c45.slice/crio-e9854b46bbe5b9e5c2c856f09b60b8e1685bbc7dc8a2006e55cfabb3cd8d0874 WatchSource:0}: Error finding container e9854b46bbe5b9e5c2c856f09b60b8e1685bbc7dc8a2006e55cfabb3cd8d0874: Status 404 returned error can't find the container with id e9854b46bbe5b9e5c2c856f09b60b8e1685bbc7dc8a2006e55cfabb3cd8d0874 Jul 07 04:24:06.089449 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.089421 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1afd5c_8a3a_47a6_bdf1_5838616f1b6f.slice/crio-301a872397ae4a5bc35aecc54859b0bd139f21d2d0692b0b2dfe68c458e4e8b3 WatchSource:0}: Error finding container 301a872397ae4a5bc35aecc54859b0bd139f21d2d0692b0b2dfe68c458e4e8b3: Status 404 returned error can't find the container with id 301a872397ae4a5bc35aecc54859b0bd139f21d2d0692b0b2dfe68c458e4e8b3 Jul 07 04:24:06.090757 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:06.090707 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f94c1df_25d3_4528_9f2b_d7b3d3dce402.slice/crio-8b2e500587b788d286e7a5c8189cf6936264211b9155381ab56a20502cc98708 WatchSource:0}: Error finding container 8b2e500587b788d286e7a5c8189cf6936264211b9155381ab56a20502cc98708: Status 404 returned error can't find the container with id 8b2e500587b788d286e7a5c8189cf6936264211b9155381ab56a20502cc98708 Jul 07 04:24:06.137409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.137368 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 04:19:05 +0000 UTC" deadline="2028-03-02 09:21:31.982455045 +0000 UTC" Jul 07 04:24:06.137409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.137406 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14500h57m25.845054136s" Jul 07 04:24:06.240376 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.240195 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" event={"ID":"be24b6de-a673-43bb-9b7b-55c29137a1f6","Type":"ContainerStarted","Data":"edc15899563df7f8eb461da6d40fde9bcdc230af4f8a60fa3d2193be83c7661a"} Jul 07 04:24:06.241181 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.241153 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerStarted","Data":"00a5621ae40a2d71f76b23a4f0a13b870f54e32a19c08eb742db9be132bebee9"} Jul 07 04:24:06.242164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.242143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v7d99" event={"ID":"c406620d-9bcd-476c-90b4-ec9e51b25c45","Type":"ContainerStarted","Data":"e9854b46bbe5b9e5c2c856f09b60b8e1685bbc7dc8a2006e55cfabb3cd8d0874"} Jul 07 04:24:06.243120 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.243100 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57qdr" event={"ID":"0134d13a-7be7-4059-ad00-f6cb9898f4e5","Type":"ContainerStarted","Data":"6b8a2bc97ccf1d50aaadbfbf57852aa51dda6eb8a02b6c94ecbc92ce7a8f8843"} Jul 07 04:24:06.244712 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.244694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-98hh5" event={"ID":"acbacb48-194a-4e8d-acff-2310e35c7b03","Type":"ContainerStarted","Data":"8f9c705a278ff3f93fbba57381fdea629de6663c0662d4bb1ac1de50554f59a6"} Jul 07 04:24:06.245786 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.245769 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gxjfc" event={"ID":"534725e6-d911-4e9d-bee9-a90ceb824a86","Type":"ContainerStarted","Data":"25f5e133de524797a610205ac3f4765aa5e5b659fb782a648548d554a39f63d4"} Jul 07 04:24:06.246762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.246743 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ttvr" event={"ID":"ababb864-9ac2-497c-b428-cff38e27581a","Type":"ContainerStarted","Data":"a02cde89fa94ed005571dc5796e2447378385ab82eebd8dab72a28dd7f0be057"} Jul 07 04:24:06.249441 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.249417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"7ebfeb482643ae23298dcd20cee13bf5","Type":"ContainerStarted","Data":"ab403e93c0c7f02e563e3987f6a944149c725175ba0f254760766a84b59a0f78"} Jul 07 04:24:06.250868 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.250849 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"8b2e500587b788d286e7a5c8189cf6936264211b9155381ab56a20502cc98708"} Jul 07 04:24:06.255678 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.255650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnslv" event={"ID":"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f","Type":"ContainerStarted","Data":"301a872397ae4a5bc35aecc54859b0bd139f21d2d0692b0b2dfe68c458e4e8b3"} Jul 07 04:24:06.263922 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.263865 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" podStartSLOduration=2.263850535 podStartE2EDuration="2.263850535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:06.263219907 +0000 UTC m=+3.930638101" watchObservedRunningTime="2026-07-07 04:24:06.263850535 +0000 UTC m=+3.931268727" Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.811544 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.811598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.811724 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.811787 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:08.81176836 +0000 UTC m=+6.479186533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.812199 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:06.814465 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.812253 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:08.812236138 +0000 UTC m=+6.479654314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:06.912788 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:06.912666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:06.912957 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.912878 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:06.912957 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.912906 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:06.912957 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.912921 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:06.913194 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:06.912982 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:08.912963821 +0000 UTC m=+6.580381996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:07.138247 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.138101 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 04:19:05 +0000 UTC" deadline="2028-02-09 10:16:09.694489883 +0000 UTC" Jul 07 04:24:07.138247 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.138164 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13973h52m2.556329442s" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.232947 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:07.233096 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.233473 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:07.233560 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.233623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:07.233752 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:07.233696 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:07.269334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.268837 2560 generic.go:356] "Generic (PLEG): container finished" podID="311c3c33db1bb18e1e66fdc4b1c32e51" containerID="0df6b307ebf434f009cbd57a30156e76a18ab67a93c718ec0c334b7291d8ccc1" exitCode=0 Jul 07 04:24:07.269334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:07.269279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"311c3c33db1bb18e1e66fdc4b1c32e51","Type":"ContainerDied","Data":"0df6b307ebf434f009cbd57a30156e76a18ab67a93c718ec0c334b7291d8ccc1"} Jul 07 04:24:08.285753 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:08.285685 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"311c3c33db1bb18e1e66fdc4b1c32e51","Type":"ContainerStarted","Data":"cabb2d6a32b85ad91ca909ba4d8d0835590abb2d75a25127e26cd088a301cd24"} Jul 07 04:24:08.303533 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:08.303121 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" podStartSLOduration=4.303086129 podStartE2EDuration="4.303086129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:08.301981235 +0000 UTC m=+5.969399430" watchObservedRunningTime="2026-07-07 04:24:08.303086129 +0000 UTC m=+5.970504322" Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:08.828612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:08.828665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.828813 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.828873 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:12.828853603 +0000 UTC m=+10.496271778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.829284 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:08.829402 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.829334 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:12.829318853 +0000 UTC m=+10.496737027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:08.929588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:08.929540 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:08.929769 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.929711 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:08.929769 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.929732 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:08.929769 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.929745 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:08.929932 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:08.929805 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:12.9297881 +0000 UTC m=+10.597206288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:09.232758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:09.232678 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:09.232909 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:09.232828 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:09.233323 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:09.233176 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:09.233323 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:09.233286 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:09.233489 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:09.233325 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:09.233489 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:09.233437 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:11.231117 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:11.231019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:11.231583 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:11.231173 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:11.231583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:11.231020 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:11.231688 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:11.231621 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:11.231688 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:11.231677 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:11.231805 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:11.231780 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:12.861866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:12.861818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:12.862336 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:12.861901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:12.862336 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.861976 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:12.862336 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.862037 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:12.862336 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.862057 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:20.862043293 +0000 UTC m=+18.529461467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:12.862336 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.862100 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:20.862081134 +0000 UTC m=+18.529499308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:12.962807 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:12.962764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:12.962980 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.962905 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:12.962980 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.962933 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:12.962980 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.962949 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:12.963175 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:12.963023 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:20.962995568 +0000 UTC m=+18.630413738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:13.231868 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:13.231998 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:13.232404 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:13.232497 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:13.232533 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:13.232657 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:13.232570 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:15.230574 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:15.230540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:15.231053 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:15.230540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:15.231053 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:15.230656 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:15.231053 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:15.230671 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:15.231053 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:15.230772 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:15.231053 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:15.230876 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:17.230185 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:17.230106 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:17.230185 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:17.230158 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:17.230765 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:17.230221 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:17.230765 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:17.230233 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:17.230765 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:17.230331 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:17.230765 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:17.230458 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:19.230574 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:19.230527 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:19.230574 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:19.230571 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:19.231135 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:19.230592 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:19.231135 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:19.230683 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:19.231135 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:19.230808 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:19.231135 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:19.230878 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:20.928406 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:20.928325 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:20.928406 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:20.928360 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:20.928959 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:20.928487 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:20.928959 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:20.928539 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:24:36.928523242 +0000 UTC m=+34.595941411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:20.928959 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:20.928488 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:20.928959 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:20.928621 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:24:36.928606706 +0000 UTC m=+34.596024894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:21.029040 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:21.028979 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:21.029199 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.029140 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:21.029199 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.029166 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:21.029199 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.029183 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:21.029343 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.029250 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:24:37.02923236 +0000 UTC m=+34.696650529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:21.231055 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:21.230937 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:21.231207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:21.231118 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:21.231207 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.231119 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:21.231292 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.231224 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:21.231292 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:21.231265 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:21.231369 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:21.231323 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:23.231253 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:23.231207 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:23.231710 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:23.231313 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:23.231710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:23.231323 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:23.231710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:23.231408 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:23.231710 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:23.231514 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:23.231710 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:23.231621 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:25.230284 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.230259 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:25.230652 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:25.230385 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:25.231092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.230798 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:25.231092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.230806 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:25.231092 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:25.230945 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:25.231092 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:25.231051 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:25.321951 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.321918 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v7d99" event={"ID":"c406620d-9bcd-476c-90b4-ec9e51b25c45","Type":"ContainerStarted","Data":"1d2c9c94c1b4a708cc935a07d25ab6b14396106d71e4feefd68a110f4a1a39c1"} Jul 07 04:24:25.323378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.323347 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-98hh5" event={"ID":"acbacb48-194a-4e8d-acff-2310e35c7b03","Type":"ContainerStarted","Data":"3f3757ec40899a22c4bd2de78d75ed0887251c2c44fbaf1eb069538b81ab3837"} Jul 07 04:24:25.324611 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.324588 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gxjfc" event={"ID":"534725e6-d911-4e9d-bee9-a90ceb824a86","Type":"ContainerStarted","Data":"8f6f95c79e7690092b77a709e9573543cd83f2ac64c3a817f7c1dd4bc1ef5b0c"} Jul 07 04:24:25.325979 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.325950 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ttvr" event={"ID":"ababb864-9ac2-497c-b428-cff38e27581a","Type":"ContainerStarted","Data":"562a4b81a38e1a546836651d43b3178564969310c37cad60470160a80b6ec39f"} Jul 07 04:24:25.327304 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.327280 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bnslv" event={"ID":"cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f","Type":"ContainerStarted","Data":"817ba6621775cbbfcaacb40cb450f4921c3bf1e8236b10c957a9224b4786b761"} Jul 07 04:24:25.328635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.328616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" event={"ID":"be24b6de-a673-43bb-9b7b-55c29137a1f6","Type":"ContainerStarted","Data":"22c04825bc03742b26db843c411e3bd5b10ba7a17dd08322fad068fbaf4a0421"} Jul 07 04:24:25.329850 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.329830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerStarted","Data":"8af832102bbb08dd21da86777f982e924d14fb2ac73e98a2b88a140cb3e67650"} Jul 07 04:24:25.342699 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.342644 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v7d99" podStartSLOduration=3.3915926020000002 podStartE2EDuration="22.342629914s" totalImagesPullingTime="18.951037312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.342319035 +0000 UTC m=+23.009737228" watchObservedRunningTime="2026-07-07 04:24:25.342629914 +0000 UTC m=+23.010048105" Jul 07 04:24:25.357216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.357168 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bnslv" podStartSLOduration=3.441202171 podStartE2EDuration="22.357154691s" totalImagesPullingTime="18.91595252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.356708258 +0000 UTC m=+23.024126450" watchObservedRunningTime="2026-07-07 04:24:25.357154691 +0000 UTC m=+23.024572882" Jul 07 04:24:25.415983 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.415918 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gxjfc" podStartSLOduration=8.063662508 podStartE2EDuration="22.41589424s" totalImagesPullingTime="14.352231732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.415460944 +0000 UTC m=+23.082879135" watchObservedRunningTime="2026-07-07 04:24:25.41589424 +0000 UTC m=+23.083312452" Jul 07 04:24:25.416633 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.416591 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-98hh5" podStartSLOduration=3.494687426 podStartE2EDuration="22.416575167s" totalImagesPullingTime="18.921887741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.398511289 +0000 UTC m=+23.065929480" watchObservedRunningTime="2026-07-07 04:24:25.416575167 +0000 UTC m=+23.083993360" Jul 07 04:24:25.429596 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.429548 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9ttvr" podStartSLOduration=3.485024252 podStartE2EDuration="22.429532788s" totalImagesPullingTime="18.944508536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:25.429395994 +0000 UTC m=+23.096814190" watchObservedRunningTime="2026-07-07 04:24:25.429532788 +0000 UTC m=+23.096950979" Jul 07 04:24:25.486995 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.486905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:25.561548 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.561356 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:25.833865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.833704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:25.959068 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:25.959044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:26.051335 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.051309 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:26.334655 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.334557 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="8af832102bbb08dd21da86777f982e924d14fb2ac73e98a2b88a140cb3e67650" exitCode=0 Jul 07 04:24:26.334655 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.334643 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"8af832102bbb08dd21da86777f982e924d14fb2ac73e98a2b88a140cb3e67650"} Jul 07 04:24:26.336578 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.336558 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57qdr" event={"ID":"0134d13a-7be7-4059-ad00-f6cb9898f4e5","Type":"ContainerStarted","Data":"cd5e8a4980b49eb74171d5ba79126db796866cc4f27056498183f714c09d95ee"} Jul 07 04:24:26.340340 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.340318 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:24:26.340773 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.340749 2560 generic.go:356] "Generic (PLEG): container finished" podID="8f94c1df-25d3-4528-9f2b-d7b3d3dce402" containerID="ba4799f9c6ee4b03d232283f5352b8763982fab2f63f15a7e0495fe199164c25" exitCode=1 Jul 07 04:24:26.340877 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.340852 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"277b9deb09d3359578a4efca915fdabb699fc3a8a0c35f8fde36a54246b373dd"} Jul 07 04:24:26.340942 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.340926 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"9a08646149b06924507d8ec80f61b709aa96966930711cfcdb404f1675804a37"} Jul 07 04:24:26.340998 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.340979 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"6411474cfb2bfa183adc0800f39c3b7297e2dcedd42d41589eb02a6ee495eb7b"} Jul 07 04:24:26.341067 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.341049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"2366278ee11e0bafe3da6f708dde3a73ac18954115974530063c518c6600383d"} Jul 07 04:24:26.341110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.341096 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerDied","Data":"ba4799f9c6ee4b03d232283f5352b8763982fab2f63f15a7e0495fe199164c25"} Jul 07 04:24:26.341158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.341144 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"e81c00c9ba1bc89a223313e22d578f4da40d8a4f9dff9c8eca2d03b2650bbbda"} Jul 07 04:24:26.370236 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.370191 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57qdr" podStartSLOduration=4.448615439 podStartE2EDuration="23.370177771s" totalImagesPullingTime="18.921562332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:26.369764392 +0000 UTC m=+24.037182594" watchObservedRunningTime="2026-07-07 04:24:26.370177771 +0000 UTC m=+24.037596004" Jul 07 04:24:26.771987 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:26.771941 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 04:24:27.074197 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.074120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.138444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.138408 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.151514 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.151404 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T04:24:26.771959373Z","UUID":"520cd253-302e-4547-9e88-8079fab5676d","Handler":null,"Name":"","Endpoint":""} Jul 07 04:24:27.153176 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.153151 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 04:24:27.153327 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.153200 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 04:24:27.208374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.208342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.230401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.230368 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:27.230401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.230391 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:27.230590 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.230373 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:27.230590 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:27.230482 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:27.230590 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:27.230547 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:27.230710 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:27.230634 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:27.308507 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.308473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.347365 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.347275 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" event={"ID":"be24b6de-a673-43bb-9b7b-55c29137a1f6","Type":"ContainerStarted","Data":"da8654e5d86e5948311d99f79d5113dc1e08aa263fdcc6d96ab82b6ac18d773f"} Jul 07 04:24:27.441863 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.441825 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.837118 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.837047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:27.925555 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:27.925520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:28.353673 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:28.353646 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:24:28.354286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:28.354254 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"7b8192e5ffda7299a1b3d57ac693dcfb18f9ca4d4670a76ceb19f44e4fa7ae04"} Jul 07 04:24:28.358534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:28.358505 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" event={"ID":"be24b6de-a673-43bb-9b7b-55c29137a1f6","Type":"ContainerStarted","Data":"1de7c16dbad286576feb3e0b4786752b58c1f0eabd59a6858440dc658786c472"} Jul 07 04:24:28.378921 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:28.378849 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g8kwx" podStartSLOduration=3.773568268 podStartE2EDuration="25.378834246s" totalImagesPullingTime="21.605265978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:28.378661659 +0000 UTC m=+26.046079852" watchObservedRunningTime="2026-07-07 04:24:28.378834246 +0000 UTC m=+26.046252439" Jul 07 04:24:29.230644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:29.230608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:29.230644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:29.230640 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:29.230842 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:29.230612 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:29.230842 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:29.230725 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:29.230842 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:29.230812 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:29.230941 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:29.230896 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:30.221726 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:30.221692 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:30.222610 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:30.222588 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:30.363798 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:30.363767 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:30.364336 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:30.364313 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gxjfc" Jul 07 04:24:30.421607 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:30.421574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:31.230959 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:31.230925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:31.231480 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:31.230925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:31.231480 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:31.230925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:31.231480 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:31.231102 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:31.231480 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:31.231181 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:31.231480 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:31.231284 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:32.371316 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.371239 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:24:32.371824 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.371608 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"43f69b88e6ba2088a75d994de4c27829d969c585bba7172be45349bb3aba0f7c"} Jul 07 04:24:32.371968 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.371948 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:32.372103 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.372078 2560 scope.go:117] "RemoveContainer" containerID="ba4799f9c6ee4b03d232283f5352b8763982fab2f63f15a7e0495fe199164c25" Jul 07 04:24:32.373708 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.373687 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="faf8195be0097a619ac1b6975c4f17bef9d4d51c86884d6076dd3b84f827b4ac" exitCode=0 Jul 07 04:24:32.373797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.373768 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"faf8195be0097a619ac1b6975c4f17bef9d4d51c86884d6076dd3b84f827b4ac"} Jul 07 04:24:32.387520 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:32.387500 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:33.004790 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.004762 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:33.098205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.098176 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:33.202489 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.202447 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:33.230974 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.230939 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:33.231170 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.231074 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:33.231170 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.231084 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:33.231291 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.231196 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:33.231291 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.231209 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:33.231291 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.231274 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:33.379526 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.379503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:24:33.379864 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.379833 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" event={"ID":"8f94c1df-25d3-4528-9f2b-d7b3d3dce402","Type":"ContainerStarted","Data":"199449e12ea66ad3f458be5eadc53e2980d1938e83cd61c8fd9273dcea95da44"} Jul 07 04:24:33.380211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.380172 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:33.380703 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.380676 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:33.400872 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.400836 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:24:33.412346 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.412286 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" podStartSLOduration=10.990071819 podStartE2EDuration="30.412267188s" totalImagesPullingTime="19.422195369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:33.410638299 +0000 UTC m=+31.078056499" watchObservedRunningTime="2026-07-07 04:24:33.412267188 +0000 UTC m=+31.079685381" Jul 07 04:24:33.670444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.670237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:33.737402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.737369 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:33.782117 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.782088 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tx8w2"] Jul 07 04:24:33.782287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.782199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:33.782349 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.782303 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:33.785955 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.785917 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zd488"] Jul 07 04:24:33.786083 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.786055 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:33.786176 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.786152 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:33.786579 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.786560 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdqmk"] Jul 07 04:24:33.786676 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.786663 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:33.786768 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:33.786750 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:33.901895 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:33.901864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:34.003161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.003071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:34.384833 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.384800 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="335842099a75f62e67ddad828ec2deb3fba0fc1111138a581c0a117abe94d9bb" exitCode=0 Jul 07 04:24:34.385435 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.384873 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"335842099a75f62e67ddad828ec2deb3fba0fc1111138a581c0a117abe94d9bb"} Jul 07 04:24:34.435686 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.435659 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:34.503145 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.503119 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:34.925131 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:34.925092 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:35.025142 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.025101 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:35.231060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.230993 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:35.231219 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.231164 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:35.231219 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:35.231179 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:35.231219 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.231194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:35.231318 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:35.231268 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:35.231353 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:35.231338 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:35.389270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.389187 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="ce1279085374610ff07f542d55d313521a5bcc71aba066e550fd0879aff6ba77" exitCode=0 Jul 07 04:24:35.389898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.389275 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"ce1279085374610ff07f542d55d313521a5bcc71aba066e550fd0879aff6ba77"} Jul 07 04:24:35.878804 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.878772 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:35.958895 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:35.958859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:36.072348 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:36.072312 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:36.172167 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:36.172073 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:36.941384 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:36.941340 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:36.947314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:36.947284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:36.947460 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:36.947327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:36.947460 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:36.947442 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:36.947550 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:36.947467 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:36.947550 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:36.947516 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret podName:5ded6da4-58bc-4b73-b3fc-d47d9114e4f2 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:08.947494368 +0000 UTC m=+66.614912542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret") pod "global-pull-secret-syncer-tx8w2" (UID: "5ded6da4-58bc-4b73-b3fc-d47d9114e4f2") : object "kube-system"/"original-pull-secret" not registered Jul 07 04:24:36.947550 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:36.947536 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs podName:72012cd3-d8fa-46a2-90b8-32e2e92c4b6a nodeName:}" failed. No retries permitted until 2026-07-07 04:25:08.947525571 +0000 UTC m=+66.614943748 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs") pod "network-metrics-daemon-mdqmk" (UID: "72012cd3-d8fa-46a2-90b8-32e2e92c4b6a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 04:24:37.048043 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:37.048012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:37.048214 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.048138 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 04:24:37.048214 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.048167 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 04:24:37.048214 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.048179 2560 projected.go:196] Error preparing data for projected volume kube-api-access-ld6gd for pod openshift-network-diagnostics/network-check-target-zd488: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:37.048381 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.048240 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd podName:f10f2cf4-fca7-407c-b162-bc708c3fd89c nodeName:}" failed. No retries permitted until 2026-07-07 04:25:09.04822209 +0000 UTC m=+66.715640277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ld6gd" (UniqueName: "kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd") pod "network-check-target-zd488" (UID: "f10f2cf4-fca7-407c-b162-bc708c3fd89c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 04:24:37.230806 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:37.230728 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:37.230966 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:37.230729 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:37.230966 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.230856 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:37.230966 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:37.230728 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:37.230966 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.230943 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:37.231150 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:37.231075 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:38.261406 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.261363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:38.454286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.454246 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:38.558907 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.558803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:38.659874 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.659838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:38.756086 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.756051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:38.973584 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:38.973546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:39.230571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:39.230428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:39.230571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:39.230428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:39.230571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:39.230546 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:39.230571 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:39.230548 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:39.230866 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:39.230665 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:39.230866 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:39.230761 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:39.266916 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:39.266886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:39.566297 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:39.566265 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:40.369585 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.369406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:40.403867 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.403833 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="e778a66fda6beac8fdf698b59d3dfde22961edfbb5b78c9d4e0473474fd323c6" exitCode=0 Jul 07 04:24:40.403867 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.403870 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"e778a66fda6beac8fdf698b59d3dfde22961edfbb5b78c9d4e0473474fd323c6"} Jul 07 04:24:40.568337 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.568303 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:40.669760 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.669652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:40.770487 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:40.770461 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:41.175545 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.175514 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:41.230618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.230580 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:41.230788 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.230625 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:41.230788 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.230702 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:41.230788 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:41.230721 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:41.230893 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:41.230787 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:41.230893 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:41.230858 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:41.408391 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.408357 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2a084a3-547a-4578-bd94-675420fca735" containerID="1963b105bed351c929791943168af21de1b303a367cd242aa1f6bd4506d019fe" exitCode=0 Jul 07 04:24:41.408772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.408404 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerDied","Data":"1963b105bed351c929791943168af21de1b303a367cd242aa1f6bd4506d019fe"} Jul 07 04:24:41.563044 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.562989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:41.664015 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.663970 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:41.965458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:41.965408 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:42.414295 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:42.414258 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" event={"ID":"b2a084a3-547a-4578-bd94-675420fca735","Type":"ContainerStarted","Data":"5bec691265c23934d54201503d4bd56821001b62baad982da9ce893180d76d51"} Jul 07 04:24:42.438927 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:42.438881 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7xfmc" podStartSLOduration=8.705713231 podStartE2EDuration="39.438864826s" totalImagesPullingTime="30.733151595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:42.437321929 +0000 UTC m=+40.104740123" watchObservedRunningTime="2026-07-07 04:24:42.438864826 +0000 UTC m=+40.106283016" Jul 07 04:24:42.483108 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:42.483077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:42.583433 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:42.583406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.118402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.118371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.217961 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.217914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.231626 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.231600 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:43.231823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.231711 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:43.231823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.231737 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:43.231823 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:43.231789 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:43.231823 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:43.231703 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:43.232059 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:43.231949 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:43.360519 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.360469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.462571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.462482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.758983 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.758902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:43.963401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:43.963372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:44.362997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:44.362967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:44.868893 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:44.868863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:44.969690 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:44.969661 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:45.233708 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:45.233639 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:45.233838 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:45.233722 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:45.233838 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:45.233726 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:45.233838 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:45.233813 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:45.233838 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:45.233832 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:45.233961 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:45.233881 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:45.358144 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:45.358110 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:45.820300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:45.820266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:46.171405 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:46.171318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:46.362128 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:46.362082 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:46.773131 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:46.773094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:47.230253 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:47.230220 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:47.230744 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:47.230230 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:47.230744 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:47.230320 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:47.230744 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:47.230332 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:47.230744 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:47.230408 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:47.230744 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:47.230469 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:47.359532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:47.359489 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:47.762773 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:47.762724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:48.176945 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:48.176912 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:48.369065 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:48.369026 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:48.565513 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:48.565487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:49.028458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:49.028379 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:49.230172 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:49.230140 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:49.230347 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:49.230153 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:49.230347 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:49.230232 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:49.230347 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:49.230151 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:49.230347 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:49.230303 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:49.230489 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:49.230388 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:51.230334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:51.230288 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:51.230334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:51.230320 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:51.230334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:51.230288 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:51.230840 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:51.230419 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:51.230840 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:51.230466 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:51.230840 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:51.230545 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:52.626984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:52.626947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:52.687177 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:52.687142 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:53.230928 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:53.230884 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:53.231135 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:53.230985 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:53.231135 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:53.230995 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:53.231135 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:53.231093 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:53.231135 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:53.231116 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:53.231296 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:53.231197 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:55.230191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.230156 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:55.230630 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.230156 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:55.230630 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:55.230269 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zd488" podUID="f10f2cf4-fca7-407c-b162-bc708c3fd89c" Jul 07 04:24:55.230630 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.230156 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:55.230630 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:55.230334 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tx8w2" podUID="5ded6da4-58bc-4b73-b3fc-d47d9114e4f2" Jul 07 04:24:55.230630 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:24:55.230421 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdqmk" podUID="72012cd3-d8fa-46a2-90b8-32e2e92c4b6a" Jul 07 04:24:55.775277 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.775233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:55.868872 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.868838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:55.972311 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:55.972211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.067234 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.067202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.128891 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.128865 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeReady" Jul 07 04:24:56.129044 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.128987 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 04:24:56.150653 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.150617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.168380 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.168347 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6df6899b44-mzhn4"] Jul 07 04:24:56.186345 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.186318 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6df6899b44-mzhn4"] Jul 07 04:24:56.186483 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.186404 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.189282 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.189254 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6jvd\"" Jul 07 04:24:56.189436 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.189301 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 04:24:56.189436 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.189339 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 04:24:56.189548 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.189471 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 04:24:56.191057 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.191037 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lzpwn"] Jul 07 04:24:56.196430 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.196408 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 04:24:56.206485 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.206463 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dqjvx"] Jul 07 04:24:56.206597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.206577 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.209333 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.209310 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jlfpx\"" Jul 07 04:24:56.209458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.209352 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 04:24:56.209458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.209589 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 04:24:56.210138 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.210126 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 04:24:56.220635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.220614 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lzpwn"] Jul 07 04:24:56.220735 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.220711 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.224136 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.224084 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 04:24:56.224402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.224387 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qn6gg\"" Jul 07 04:24:56.224497 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.224441 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 04:24:56.225756 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.225733 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dqjvx"] Jul 07 04:24:56.290092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-bound-sa-token\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290110 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7a010-494d-478f-8ece-c4cf1debc5df-tmp-dir\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290138 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef7a010-494d-478f-8ece-c4cf1debc5df-config-volume\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290199 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v8l\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-kube-api-access-t5v8l\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290237 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef7a010-494d-478f-8ece-c4cf1debc5df-metrics-tls\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-trusted-ca\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-installation-pull-secrets\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-image-registry-private-configuration\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afa7c157-2a1c-4421-8730-eae14f2fc659-cert\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290388 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvdh\" (UniqueName: \"kubernetes.io/projected/afa7c157-2a1c-4421-8730-eae14f2fc659-kube-api-access-tlvdh\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-certificates\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290426 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-tls\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290447 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-ca-trust-extracted\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.290501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.290480 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwg4n\" (UniqueName: \"kubernetes.io/projected/bef7a010-494d-478f-8ece-c4cf1debc5df-kube-api-access-qwg4n\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.347285 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.347254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.391047 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.390986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-image-registry-private-configuration\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391047 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afa7c157-2a1c-4421-8730-eae14f2fc659-cert\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.391293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391062 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvdh\" (UniqueName: \"kubernetes.io/projected/afa7c157-2a1c-4421-8730-eae14f2fc659-kube-api-access-tlvdh\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.391293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-certificates\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-tls\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391256 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-ca-trust-extracted\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391274 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwg4n\" (UniqueName: \"kubernetes.io/projected/bef7a010-494d-478f-8ece-c4cf1debc5df-kube-api-access-qwg4n\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-bound-sa-token\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391345 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7a010-494d-478f-8ece-c4cf1debc5df-tmp-dir\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391381 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef7a010-494d-478f-8ece-c4cf1debc5df-config-volume\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v8l\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-kube-api-access-t5v8l\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef7a010-494d-478f-8ece-c4cf1debc5df-metrics-tls\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-trusted-ca\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391508 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-installation-pull-secrets\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391611 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-ca-trust-extracted\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.391952 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.391931 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-certificates\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.392475 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.392449 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bef7a010-494d-478f-8ece-c4cf1debc5df-config-volume\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.392764 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.392741 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef7a010-494d-478f-8ece-c4cf1debc5df-tmp-dir\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.392897 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.392854 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-trusted-ca\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.395428 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.395399 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bef7a010-494d-478f-8ece-c4cf1debc5df-metrics-tls\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.395524 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.395428 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-installation-pull-secrets\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.395524 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.395432 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-image-registry-private-configuration\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.395524 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.395457 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/afa7c157-2a1c-4421-8730-eae14f2fc659-cert\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.395632 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.395530 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-registry-tls\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.405206 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.405175 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwg4n\" (UniqueName: \"kubernetes.io/projected/bef7a010-494d-478f-8ece-c4cf1debc5df-kube-api-access-qwg4n\") pod \"dns-default-dqjvx\" (UID: \"bef7a010-494d-478f-8ece-c4cf1debc5df\") " pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.408600 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.408573 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvdh\" (UniqueName: \"kubernetes.io/projected/afa7c157-2a1c-4421-8730-eae14f2fc659-kube-api-access-tlvdh\") pod \"ingress-canary-lzpwn\" (UID: \"afa7c157-2a1c-4421-8730-eae14f2fc659\") " pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.414720 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.414699 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-bound-sa-token\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.415074 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.415055 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v8l\" (UniqueName: \"kubernetes.io/projected/0855b731-2cf2-43d4-8f02-90ba61c9c7f7-kube-api-access-t5v8l\") pod \"image-registry-6df6899b44-mzhn4\" (UID: \"0855b731-2cf2-43d4-8f02-90ba61c9c7f7\") " pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.438664 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.438627 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.497299 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.497201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:56.516352 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.516316 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lzpwn" Jul 07 04:24:56.526854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.526823 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:56.529511 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.529492 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:56.707134 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.707099 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lzpwn"] Jul 07 04:24:56.711612 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:56.711586 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa7c157_2a1c_4421_8730_eae14f2fc659.slice/crio-7d772a57df4eddbe5acb8eab3c143e40bf749cb15f9cd3b8cf59e69d2ae33224 WatchSource:0}: Error finding container 7d772a57df4eddbe5acb8eab3c143e40bf749cb15f9cd3b8cf59e69d2ae33224: Status 404 returned error can't find the container with id 7d772a57df4eddbe5acb8eab3c143e40bf749cb15f9cd3b8cf59e69d2ae33224 Jul 07 04:24:56.725258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.725232 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6df6899b44-mzhn4"] Jul 07 04:24:56.728280 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:56.728253 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0855b731_2cf2_43d4_8f02_90ba61c9c7f7.slice/crio-7d0f9f57f340859d374fc0f3bd26ae4d9696f5fc52f7b8d8f0c6ec4905ffc721 WatchSource:0}: Error finding container 7d0f9f57f340859d374fc0f3bd26ae4d9696f5fc52f7b8d8f0c6ec4905ffc721: Status 404 returned error can't find the container with id 7d0f9f57f340859d374fc0f3bd26ae4d9696f5fc52f7b8d8f0c6ec4905ffc721 Jul 07 04:24:56.736946 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:56.736920 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dqjvx"] Jul 07 04:24:56.740619 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:24:56.740591 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef7a010_494d_478f_8ece_c4cf1debc5df.slice/crio-43eed9f08c913bc65cc6cf1f77e507d3848e0b11f9fb5799d6606a68af09994b WatchSource:0}: Error finding container 43eed9f08c913bc65cc6cf1f77e507d3848e0b11f9fb5799d6606a68af09994b: Status 404 returned error can't find the container with id 43eed9f08c913bc65cc6cf1f77e507d3848e0b11f9fb5799d6606a68af09994b Jul 07 04:24:57.067349 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.067318 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:57.138193 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.138156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:57.230893 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.230859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:24:57.231105 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.231029 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:24:57.231105 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.231069 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:24:57.234936 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.234907 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 04:24:57.235114 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.234938 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 04:24:57.235114 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.234959 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 04:24:57.235114 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.235032 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-krw7t\"" Jul 07 04:24:57.235114 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.235110 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krb48\"" Jul 07 04:24:57.235284 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.235231 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 04:24:57.391338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.391252 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:57.448752 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.448713 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dqjvx" event={"ID":"bef7a010-494d-478f-8ece-c4cf1debc5df","Type":"ContainerStarted","Data":"43eed9f08c913bc65cc6cf1f77e507d3848e0b11f9fb5799d6606a68af09994b"} Jul 07 04:24:57.450412 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.450384 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lzpwn" event={"ID":"afa7c157-2a1c-4421-8730-eae14f2fc659","Type":"ContainerStarted","Data":"7d772a57df4eddbe5acb8eab3c143e40bf749cb15f9cd3b8cf59e69d2ae33224"} Jul 07 04:24:57.452560 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.452533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" event={"ID":"0855b731-2cf2-43d4-8f02-90ba61c9c7f7","Type":"ContainerStarted","Data":"b2dc8b1a528fcf30c7b2b3c54d79c6d9838eb6934af501ec604e6f7d76546b2f"} Jul 07 04:24:57.452694 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.452580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" event={"ID":"0855b731-2cf2-43d4-8f02-90ba61c9c7f7","Type":"ContainerStarted","Data":"7d0f9f57f340859d374fc0f3bd26ae4d9696f5fc52f7b8d8f0c6ec4905ffc721"} Jul 07 04:24:57.452694 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.452658 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:24:57.496962 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.496927 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:57.598851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.598800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:57.903603 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:57.903564 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.003366 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.003331 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.110373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.110345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.304735 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.304706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.404581 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.404554 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.460266 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.460197 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dqjvx" event={"ID":"bef7a010-494d-478f-8ece-c4cf1debc5df","Type":"ContainerStarted","Data":"7fd95ffc408355cf02e6fa25e2a63a13b1003ea8cc82cec4c8ba0d6b48452938"} Jul 07 04:24:58.713563 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.713532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:58.909225 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:58.909188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:59.008807 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.008710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:59.464530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.464490 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lzpwn" event={"ID":"afa7c157-2a1c-4421-8730-eae14f2fc659","Type":"ContainerStarted","Data":"cd6ac41615b17c177af34acbe712d6c6da5e6adc03e077d91f5f43db869809c3"} Jul 07 04:24:59.466168 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.466145 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dqjvx" event={"ID":"bef7a010-494d-478f-8ece-c4cf1debc5df","Type":"ContainerStarted","Data":"51381625c9c64206a1ac1cec68994c1bc8e0ca401504764975eecaadb5c82943"} Jul 07 04:24:59.466303 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.466288 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dqjvx" Jul 07 04:24:59.480613 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.480556 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" podStartSLOduration=4.48053503 podStartE2EDuration="4.48053503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:57.473494998 +0000 UTC m=+55.140913190" watchObservedRunningTime="2026-07-07 04:24:59.48053503 +0000 UTC m=+57.147953222" Jul 07 04:24:59.481143 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.481116 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lzpwn" podStartSLOduration=1.004802578 podStartE2EDuration="3.48110851s" totalImagesPullingTime="2.476305932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:59.480103386 +0000 UTC m=+57.147521576" watchObservedRunningTime="2026-07-07 04:24:59.48110851 +0000 UTC m=+57.148526702" Jul 07 04:24:59.501473 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.501411 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dqjvx" podStartSLOduration=1.94008892 podStartE2EDuration="3.501391389s" totalImagesPullingTime="1.561302469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:24:59.501082366 +0000 UTC m=+57.168500559" watchObservedRunningTime="2026-07-07 04:24:59.501391389 +0000 UTC m=+57.168809582" Jul 07 04:24:59.619681 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.619633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:59.719039 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.718940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:24:59.771685 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:24:59.771654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:01.941953 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:01.941910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:02.008715 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.008680 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:02.080656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.080626 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:02.231803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.231731 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k6hbj"] Jul 07 04:25:02.261377 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.261343 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6hbj"] Jul 07 04:25:02.261535 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.261470 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.264182 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.264155 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 04:25:02.264182 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.264158 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 04:25:02.264776 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.264761 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwn6d\"" Jul 07 04:25:02.265359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.265340 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 04:25:02.265466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.265341 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 04:25:02.283817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.283775 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:02.341128 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.341093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0683736-82a6-4cc2-badf-9f3228d7193d-crio-socket\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.341128 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.341131 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0683736-82a6-4cc2-badf-9f3228d7193d-data-volume\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.341360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.341161 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxgs\" (UniqueName: \"kubernetes.io/projected/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-api-access-mnxgs\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.341360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.341281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.341360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.341326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.346286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.346263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:02.442263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxgs\" (UniqueName: \"kubernetes.io/projected/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-api-access-mnxgs\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442329 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0683736-82a6-4cc2-badf-9f3228d7193d-crio-socket\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442420 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0683736-82a6-4cc2-badf-9f3228d7193d-data-volume\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442666 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.442489 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:02.442666 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.442524 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:02.442666 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.442582 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:02.942562145 +0000 UTC m=+60.609980328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:02.442666 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.442598 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:02.942591104 +0000 UTC m=+60.610009274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : configmap "kube-rbac-proxy" not found Jul 07 04:25:02.442666 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442599 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0683736-82a6-4cc2-badf-9f3228d7193d-crio-socket\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.442886 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.442777 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0683736-82a6-4cc2-badf-9f3228d7193d-data-volume\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.454174 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.454148 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxgs\" (UniqueName: \"kubernetes.io/projected/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-api-access-mnxgs\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.945555 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.945514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.945967 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:02.945566 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:02.945967 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.945647 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:02.945967 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.945736 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:03.945713003 +0000 UTC m=+61.613131195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : configmap "kube-rbac-proxy" not found Jul 07 04:25:02.945967 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.945738 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:02.945967 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:02.945780 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:03.945769708 +0000 UTC m=+61.613187881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:03.324973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:03.324946 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dqjvx_bef7a010-494d-478f-8ece-c4cf1debc5df/dns/0.log" Jul 07 04:25:03.524444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:03.524389 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dqjvx_bef7a010-494d-478f-8ece-c4cf1debc5df/kube-rbac-proxy/0.log" Jul 07 04:25:03.951234 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:03.951180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:03.951234 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:03.951243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:03.951686 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:03.951309 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:03.951686 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:03.951356 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:03.951686 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:03.951376 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:05.951360407 +0000 UTC m=+63.618778576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : configmap "kube-rbac-proxy" not found Jul 07 04:25:03.951686 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:03.951397 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:05.951386133 +0000 UTC m=+63.618804304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:04.082659 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:04.082625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:04.124116 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:04.124095 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnslv_cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f/dns-node-resolver/0.log" Jul 07 04:25:04.183271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:04.183216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:04.927400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:04.927369 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6df6899b44-mzhn4_0855b731-2cf2-43d4-8f02-90ba61c9c7f7/registry/0.log" Jul 07 04:25:05.106985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.106953 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.170919 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.170892 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.327554 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.327528 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ttvr_ababb864-9ac2-497c-b428-cff38e27581a/node-ca/0.log" Jul 07 04:25:05.364537 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.364503 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.399577 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.399553 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vdn94" Jul 07 04:25:05.567153 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.567120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.623993 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.623914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.952290 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.952193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:05.968360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.968339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:05.968431 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:05.968416 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:05.968497 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:05.968485 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 04:25:05.968544 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:05.968534 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:09.968522063 +0000 UTC m=+67.635940233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : configmap "kube-rbac-proxy" not found Jul 07 04:25:05.968585 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:05.968488 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 04:25:05.968634 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:05.968623 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls podName:d0683736-82a6-4cc2-badf-9f3228d7193d nodeName:}" failed. No retries permitted until 2026-07-07 04:25:09.968610418 +0000 UTC m=+67.636028589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k6hbj" (UID: "d0683736-82a6-4cc2-badf-9f3228d7193d") : secret "insights-runtime-extractor-tls" not found Jul 07 04:25:06.052680 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:06.052650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:06.124636 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:06.124611 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lzpwn_afa7c157-2a1c-4421-8730-eae14f2fc659/serve-healthcheck-canary/0.log" Jul 07 04:25:07.129409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.129373 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:25:07.131880 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.131865 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.136136 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136107 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 04:25:07.136475 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136454 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 04:25:07.136632 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136614 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 04:25:07.136829 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136462 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 04:25:07.136935 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136917 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 04:25:07.137144 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.136556 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 04:25:07.137260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.137243 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 04:25:07.137550 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.137521 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n4kp4\"" Jul 07 04:25:07.141343 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.141316 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:25:07.176401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176344 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.176401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwsw\" (UniqueName: \"kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.176589 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176551 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.176633 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176585 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.176669 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176642 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.176710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.176669 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277870 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277870 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277945 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277924 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.277990 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.277958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwsw\" (UniqueName: \"kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.278509 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.278487 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.278644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.278505 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.278644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.278589 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.280235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.280207 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.280348 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.280281 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.286309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.286291 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwsw\" (UniqueName: \"kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw\") pod \"console-75bd8b9dc4-rdjlc\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.444561 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.444458 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:07.570346 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:07.570313 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:25:07.573048 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:07.573020 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbf9c7e_348c_428d_a983_2b3a92769d5b.slice/crio-724c27004982896a3018c97d0225d30e95e83b2664f270d1112f36e07a494649 WatchSource:0}: Error finding container 724c27004982896a3018c97d0225d30e95e83b2664f270d1112f36e07a494649: Status 404 returned error can't find the container with id 724c27004982896a3018c97d0225d30e95e83b2664f270d1112f36e07a494649 Jul 07 04:25:08.490165 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.490123 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd8b9dc4-rdjlc" event={"ID":"3fbf9c7e-348c-428d-a983-2b3a92769d5b","Type":"ContainerStarted","Data":"724c27004982896a3018c97d0225d30e95e83b2664f270d1112f36e07a494649"} Jul 07 04:25:08.952015 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.951975 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:08.992363 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.992311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:25:08.992363 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.992364 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:25:08.998177 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.998153 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 04:25:08.998334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:08.998314 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 04:25:09.005390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.005359 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72012cd3-d8fa-46a2-90b8-32e2e92c4b6a-metrics-certs\") pod \"network-metrics-daemon-mdqmk\" (UID: \"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a\") " pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:25:09.005558 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.005395 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ded6da4-58bc-4b73-b3fc-d47d9114e4f2-original-pull-secret\") pod \"global-pull-secret-syncer-tx8w2\" (UID: \"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2\") " pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:25:09.092887 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.092851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:25:09.095901 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.095876 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 04:25:09.106407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.106382 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 04:25:09.116571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.116545 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6gd\" (UniqueName: \"kubernetes.io/projected/f10f2cf4-fca7-407c-b162-bc708c3fd89c-kube-api-access-ld6gd\") pod \"network-check-target-zd488\" (UID: \"f10f2cf4-fca7-407c-b162-bc708c3fd89c\") " pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:25:09.245825 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.245750 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-krb48\"" Jul 07 04:25:09.250271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.250245 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tx8w2" Jul 07 04:25:09.253378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.253353 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdqmk" Jul 07 04:25:09.259481 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.259454 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-krw7t\"" Jul 07 04:25:09.266976 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.266955 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:25:09.345516 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.345397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:09.416614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.416445 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tx8w2"] Jul 07 04:25:09.439957 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.439929 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdqmk"] Jul 07 04:25:09.461862 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.461828 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zd488"] Jul 07 04:25:09.471697 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.471666 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dqjvx" Jul 07 04:25:09.565762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.565728 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:09.630959 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:09.630913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:10.002047 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.001939 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:10.002047 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.001981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:10.002595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.002574 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0683736-82a6-4cc2-badf-9f3228d7193d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:10.004520 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.004491 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0683736-82a6-4cc2-badf-9f3228d7193d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k6hbj\" (UID: \"d0683736-82a6-4cc2-badf-9f3228d7193d\") " pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:10.007310 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.007275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:10.073518 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.073470 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwn6d\"" Jul 07 04:25:10.081809 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.081777 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k6hbj" Jul 07 04:25:10.108259 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.108217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:10.175707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.175674 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:10.508019 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:10.507968 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72012cd3_d8fa_46a2_90b8_32e2e92c4b6a.slice/crio-c25adae1f2754398e18df9700e54044268d515ed901c9f483c0600b807860a4b WatchSource:0}: Error finding container c25adae1f2754398e18df9700e54044268d515ed901c9f483c0600b807860a4b: Status 404 returned error can't find the container with id c25adae1f2754398e18df9700e54044268d515ed901c9f483c0600b807860a4b Jul 07 04:25:10.508655 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:10.508622 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10f2cf4_fca7_407c_b162_bc708c3fd89c.slice/crio-73d06a17cdc1c2ee0a0a191258bab94e956721d8ce8a6a09c4723f6bbc5fd7ab WatchSource:0}: Error finding container 73d06a17cdc1c2ee0a0a191258bab94e956721d8ce8a6a09c4723f6bbc5fd7ab: Status 404 returned error can't find the container with id 73d06a17cdc1c2ee0a0a191258bab94e956721d8ce8a6a09c4723f6bbc5fd7ab Jul 07 04:25:10.509605 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:10.509582 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ded6da4_58bc_4b73_b3fc_d47d9114e4f2.slice/crio-8fefa627da002ccb27b029ea33163763a0774629a42b69f87380702cdaafd12c WatchSource:0}: Error finding container 8fefa627da002ccb27b029ea33163763a0774629a42b69f87380702cdaafd12c: Status 404 returned error can't find the container with id 8fefa627da002ccb27b029ea33163763a0774629a42b69f87380702cdaafd12c Jul 07 04:25:10.640623 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:10.640591 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k6hbj"] Jul 07 04:25:10.644379 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:10.644329 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0683736_82a6_4cc2_badf_9f3228d7193d.slice/crio-19971373f2a684c29ebd42a5857820d8e96ed5873a2e58961bb07c0df310b995 WatchSource:0}: Error finding container 19971373f2a684c29ebd42a5857820d8e96ed5873a2e58961bb07c0df310b995: Status 404 returned error can't find the container with id 19971373f2a684c29ebd42a5857820d8e96ed5873a2e58961bb07c0df310b995 Jul 07 04:25:11.504213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.503353 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd8b9dc4-rdjlc" event={"ID":"3fbf9c7e-348c-428d-a983-2b3a92769d5b","Type":"ContainerStarted","Data":"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17"} Jul 07 04:25:11.506120 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.506034 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6hbj" event={"ID":"d0683736-82a6-4cc2-badf-9f3228d7193d","Type":"ContainerStarted","Data":"19971373f2a684c29ebd42a5857820d8e96ed5873a2e58961bb07c0df310b995"} Jul 07 04:25:11.509198 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.509167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdqmk" event={"ID":"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a","Type":"ContainerStarted","Data":"c25adae1f2754398e18df9700e54044268d515ed901c9f483c0600b807860a4b"} Jul 07 04:25:11.511583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.511553 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zd488" event={"ID":"f10f2cf4-fca7-407c-b162-bc708c3fd89c","Type":"ContainerStarted","Data":"73d06a17cdc1c2ee0a0a191258bab94e956721d8ce8a6a09c4723f6bbc5fd7ab"} Jul 07 04:25:11.512930 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.512903 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tx8w2" event={"ID":"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2","Type":"ContainerStarted","Data":"8fefa627da002ccb27b029ea33163763a0774629a42b69f87380702cdaafd12c"} Jul 07 04:25:11.527725 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:11.527168 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75bd8b9dc4-rdjlc" podStartSLOduration=1.2650514 podStartE2EDuration="4.527148798s" totalImagesPullingTime="3.262097398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:11.525912702 +0000 UTC m=+69.193330893" watchObservedRunningTime="2026-07-07 04:25:11.527148798 +0000 UTC m=+69.194566991" Jul 07 04:25:12.166539 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.166490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:12.266597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.266517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:12.520257 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.520156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdqmk" event={"ID":"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a","Type":"ContainerStarted","Data":"ae38ebb520b1267056e2f0f49e1d19b70d99c28750a06b4a5b854bc95bd4e958"} Jul 07 04:25:12.520257 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.520236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdqmk" event={"ID":"72012cd3-d8fa-46a2-90b8-32e2e92c4b6a","Type":"ContainerStarted","Data":"3f8f4d61658c03dd29627beb6e37cc29e7e642cde5b083f29f92e7b4fee9a06a"} Jul 07 04:25:12.539407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.538687 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mdqmk" podStartSLOduration=68.101085415 podStartE2EDuration="1m9.538664861s" totalImagesPullingTime="1.437579446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:12.538044423 +0000 UTC m=+70.205462613" watchObservedRunningTime="2026-07-07 04:25:12.538664861 +0000 UTC m=+70.206083055" Jul 07 04:25:12.641566 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.641510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:12.714435 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:12.714356 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:14.633223 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:14.633180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:14.717538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:14.717495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:15.984290 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:15.984251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:16.078638 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:16.078602 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:16.503298 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:16.503260 2560 patch_prober.go:28] interesting pod/image-registry-6df6899b44-mzhn4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 04:25:16.503473 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:16.503321 2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" podUID="0855b731-2cf2-43d4-8f02-90ba61c9c7f7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 04:25:17.445464 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.445416 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:17.445860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.445480 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:17.450456 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.450432 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:17.543143 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.543058 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zd488" event={"ID":"f10f2cf4-fca7-407c-b162-bc708c3fd89c","Type":"ContainerStarted","Data":"8d9801ccdc01613a9ec528743d9e9b08f3a950b3a9fb52778d0c6e5923de4eec"} Jul 07 04:25:17.543309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.543182 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:25:17.544597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.544570 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tx8w2" event={"ID":"5ded6da4-58bc-4b73-b3fc-d47d9114e4f2","Type":"ContainerStarted","Data":"5de1be137c3b67d950e01ca35c0cbfb556bfec5fa18a9d3b88a965c444ab8f6e"} Jul 07 04:25:17.545946 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.545930 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6hbj" event={"ID":"d0683736-82a6-4cc2-badf-9f3228d7193d","Type":"ContainerStarted","Data":"0410db803d06cb3601a9f2db78caa365a5e9fd7e09a4ec29028fc1bd906e05c8"} Jul 07 04:25:17.549712 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.549696 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:25:17.560384 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.560285 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zd488" podStartSLOduration=67.793764986 podStartE2EDuration="1m14.560268775s" totalImagesPullingTime="6.766503789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:17.560126564 +0000 UTC m=+75.227544765" watchObservedRunningTime="2026-07-07 04:25:17.560268775 +0000 UTC m=+75.227686945" Jul 07 04:25:17.595793 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:17.595740 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tx8w2" podStartSLOduration=67.829094901 podStartE2EDuration="1m14.59572508s" totalImagesPullingTime="6.766630179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:24:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:17.594423191 +0000 UTC m=+75.261841382" watchObservedRunningTime="2026-07-07 04:25:17.59572508 +0000 UTC m=+75.263143270" Jul 07 04:25:18.287330 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.287225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:18.387800 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.387769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:18.457132 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.457088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:18.465289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.465265 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6df6899b44-mzhn4" Jul 07 04:25:18.552321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.552207 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6hbj" event={"ID":"d0683736-82a6-4cc2-badf-9f3228d7193d","Type":"ContainerStarted","Data":"08cae17c530f46799079e99b5af2227e2742e96683dfb848e6e5982393a7c832"} Jul 07 04:25:18.644084 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.644049 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:18.791017 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.790978 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:18.880757 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:18.880718 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:19.217070 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:19.216964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:19.316151 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:19.316117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:19.385058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:19.385024 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:20.561676 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:20.561580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k6hbj" event={"ID":"d0683736-82a6-4cc2-badf-9f3228d7193d","Type":"ContainerStarted","Data":"99e3266b85ba7cbaabdef48408b99f6941fb956becfb1ceb0793021a38c2eb0f"} Jul 07 04:25:20.582741 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:20.582693 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k6hbj" podStartSLOduration=9.113222717 podStartE2EDuration="18.582677862s" totalImagesPullingTime="9.469455145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 04:25:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:20.582339272 +0000 UTC m=+78.249757463" watchObservedRunningTime="2026-07-07 04:25:20.582677862 +0000 UTC m=+78.250096052" Jul 07 04:25:21.249737 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.249706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:21.503132 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.503034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:21.542865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.542837 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lvpvl"] Jul 07 04:25:21.544975 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.544960 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.547759 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.547737 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 04:25:21.548060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.548042 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8825k\"" Jul 07 04:25:21.548060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.548043 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 04:25:21.548195 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.548072 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 04:25:21.548311 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.548298 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 04:25:21.549180 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.549165 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 04:25:21.549248 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.549210 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 04:25:21.590772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.590743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:21.685876 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.685836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:21.688869 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.688849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.688932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.688885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-textfile\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.688983 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.688926 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689043 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-metrics-client-ca\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689105 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689091 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-root\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689172 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-sys\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689200 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-wtmp\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4m4n\" (UniqueName: \"kubernetes.io/projected/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-kube-api-access-t4m4n\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.689316 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.689297 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.787851 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.787743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:21.789765 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-textfile\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.789829 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789781 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.789829 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789804 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-metrics-client-ca\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.789894 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789828 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-root\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.789894 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789889 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-root\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.789978 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789943 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-sys\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.789996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-wtmp\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790099 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4m4n\" (UniqueName: \"kubernetes.io/projected/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-kube-api-access-t4m4n\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790226 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790148 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-textfile\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790226 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790052 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-sys\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790226 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790155 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-wtmp\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790339 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:21.790238 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 04:25:21.790339 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:21.790292 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls podName:3cad0bc2-8287-4f29-ae47-6a53f4ed5b82 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:22.290275279 +0000 UTC m=+79.957693448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls") pod "node-exporter-lvpvl" (UID: "3cad0bc2-8287-4f29-ae47-6a53f4ed5b82") : secret "node-exporter-tls" not found Jul 07 04:25:21.790460 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790441 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-metrics-client-ca\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.790593 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.790576 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.792161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.792144 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.801941 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.801916 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4m4n\" (UniqueName: \"kubernetes.io/projected/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-kube-api-access-t4m4n\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:21.885067 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:21.885031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.057312 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.057232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.275748 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.275713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.293828 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.293803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:22.296011 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.295984 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cad0bc2-8287-4f29-ae47-6a53f4ed5b82-node-exporter-tls\") pod \"node-exporter-lvpvl\" (UID: \"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82\") " pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:22.454256 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.454228 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvpvl" Jul 07 04:25:22.488814 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.488776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.566709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.566670 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.569887 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.569857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvpvl" event={"ID":"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82","Type":"ContainerStarted","Data":"1dad89565fcd62b770d52421e9c29e9eaed82b0e7fdc62755b1bb22f6b986a8b"} Jul 07 04:25:22.571359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.571340 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:25:22.574211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.574197 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.577827 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.577807 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 04:25:22.578135 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578119 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 04:25:22.578683 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578666 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-njbx4\"" Jul 07 04:25:22.578909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578890 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 04:25:22.579037 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578918 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 04:25:22.579037 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578951 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 04:25:22.579037 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.578959 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 04:25:22.579234 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.579217 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 04:25:22.579644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.579625 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 04:25:22.579781 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.579648 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 04:25:22.593378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.593350 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:25:22.688509 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.688475 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:22.695861 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695838 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695908 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695924 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695942 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.695973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695962 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695983 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.695997 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.696054 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.696088 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jg6\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.696107 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.696205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.696155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797605 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797605 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797555 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797605 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797605 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797623 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:22.797781 2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797830 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jg6\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.797873 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:22.797850 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls podName:06ca606a-1106-4d1b-9682-a6f1dc684dd2 nodeName:}" failed. No retries permitted until 2026-07-07 04:25:23.297831417 +0000 UTC m=+80.965249588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2") : secret "alertmanager-main-tls" not found Jul 07 04:25:22.798300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797891 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.798300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797933 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.798300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.798300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.797987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.798300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.798071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.799803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.798425 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.799803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.798690 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.799803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.799538 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.800657 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.800627 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.800759 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.800724 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.800821 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.800782 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.801382 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.801360 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.801513 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.801499 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.801564 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.801510 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.802261 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.802240 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.802442 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.802426 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:22.809184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:22.809154 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jg6\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:23.301320 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.301280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:23.303897 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.303871 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:23.337573 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.337539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:23.483822 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.483783 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:25:23.489499 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.489475 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-697c7f855-rc92g"] Jul 07 04:25:23.492656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.492629 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.497589 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.497568 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 04:25:23.497837 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.497817 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 04:25:23.499261 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.499216 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ffktna54dou78\"" Jul 07 04:25:23.499703 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.499682 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 04:25:23.499973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.499955 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 04:25:23.500659 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.500223 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pn6xt\"" Jul 07 04:25:23.500659 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.500427 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 04:25:23.518108 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.517996 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-697c7f855-rc92g"] Jul 07 04:25:23.603214 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603188 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603261 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603303 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-grpc-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtb5\" (UniqueName: \"kubernetes.io/projected/c1db23cf-71b6-4293-8837-21e139df2964-kube-api-access-nbtb5\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603532 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.603804 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.603580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1db23cf-71b6-4293-8837-21e139df2964-metrics-client-ca\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.657014 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.656976 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:25:23.659358 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:23.659334 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ca606a_1106_4d1b_9682_a6f1dc684dd2.slice/crio-d687b53e77376d410708ee6241defd79a2dbe2f02f4296d0c1368796e06861fa WatchSource:0}: Error finding container d687b53e77376d410708ee6241defd79a2dbe2f02f4296d0c1368796e06861fa: Status 404 returned error can't find the container with id d687b53e77376d410708ee6241defd79a2dbe2f02f4296d0c1368796e06861fa Jul 07 04:25:23.704201 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704176 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtb5\" (UniqueName: \"kubernetes.io/projected/c1db23cf-71b6-4293-8837-21e139df2964-kube-api-access-nbtb5\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704340 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704210 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704340 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704241 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704340 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704258 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1db23cf-71b6-4293-8837-21e139df2964-metrics-client-ca\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704406 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704439 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.704515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.704471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-grpc-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.705064 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.705041 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1db23cf-71b6-4293-8837-21e139df2964-metrics-client-ca\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.706877 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.706846 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.706977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.706881 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.707126 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.707109 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-grpc-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.707213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.707153 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.707277 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.707251 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.707308 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.707293 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c1db23cf-71b6-4293-8837-21e139df2964-secret-thanos-querier-tls\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.713162 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.713141 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtb5\" (UniqueName: \"kubernetes.io/projected/c1db23cf-71b6-4293-8837-21e139df2964-kube-api-access-nbtb5\") pod \"thanos-querier-697c7f855-rc92g\" (UID: \"c1db23cf-71b6-4293-8837-21e139df2964\") " pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.803215 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.803175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:23.930402 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:23.930372 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-697c7f855-rc92g"] Jul 07 04:25:23.934553 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:23.934517 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1db23cf_71b6_4293_8837_21e139df2964.slice/crio-cced1968c771264dd2d61ae417fae4384c09c6b8c02528b9eee092a1c91682a8 WatchSource:0}: Error finding container cced1968c771264dd2d61ae417fae4384c09c6b8c02528b9eee092a1c91682a8: Status 404 returned error can't find the container with id cced1968c771264dd2d61ae417fae4384c09c6b8c02528b9eee092a1c91682a8 Jul 07 04:25:24.480590 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.480416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:24.581067 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.581035 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:24.581909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.581880 2560 generic.go:356] "Generic (PLEG): container finished" podID="3cad0bc2-8287-4f29-ae47-6a53f4ed5b82" containerID="1475aac08bb6595ed6314d6f72f6840d75417b8a225edbcff2de570480ed6488" exitCode=0 Jul 07 04:25:24.582075 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.581962 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvpvl" event={"ID":"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82","Type":"ContainerDied","Data":"1475aac08bb6595ed6314d6f72f6840d75417b8a225edbcff2de570480ed6488"} Jul 07 04:25:24.584055 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.583508 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"cced1968c771264dd2d61ae417fae4384c09c6b8c02528b9eee092a1c91682a8"} Jul 07 04:25:24.585237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.585201 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"d687b53e77376d410708ee6241defd79a2dbe2f02f4296d0c1368796e06861fa"} Jul 07 04:25:24.657262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.657222 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:24.987251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:24.987206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:25.058358 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.058324 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:25.590968 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.590934 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvpvl" event={"ID":"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82","Type":"ContainerStarted","Data":"14c31d2df69a96fda11a027fbddb8473282201f10f6947ea6ba489f97e8e3f01"} Jul 07 04:25:25.591153 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.590987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvpvl" event={"ID":"3cad0bc2-8287-4f29-ae47-6a53f4ed5b82","Type":"ContainerStarted","Data":"8771396f4d945138e0a196eae933d84fe678397f550ae04a0e4c2f261aab7895"} Jul 07 04:25:25.592753 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.592725 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621" exitCode=0 Jul 07 04:25:25.592881 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.592798 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621"} Jul 07 04:25:25.614190 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:25.614146 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lvpvl" podStartSLOduration=3.450029294 podStartE2EDuration="4.614129459s" totalImagesPullingTime="1.100752489s" totalInitContainerRuntime="63.347676ms" isStatefulPod=true podCreationTimestamp="2026-07-07 04:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:25.612543584 +0000 UTC m=+83.279961777" watchObservedRunningTime="2026-07-07 04:25:25.614129459 +0000 UTC m=+83.281547651" Jul 07 04:25:26.300883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.300834 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:26.304159 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.303901 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m"] Jul 07 04:25:26.307317 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.307300 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.310055 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.310033 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j7j9m\"" Jul 07 04:25:26.310354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.310327 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 04:25:26.315912 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.315841 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m"] Jul 07 04:25:26.386110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.386028 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:26.428254 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.428221 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9181f5b-4c52-49ca-bb5d-a283bdd23769-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.428482 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.428270 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zhf\" (UniqueName: \"kubernetes.io/projected/b9181f5b-4c52-49ca-bb5d-a283bdd23769-kube-api-access-h5zhf\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.483386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.483358 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:26.529598 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.529557 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9181f5b-4c52-49ca-bb5d-a283bdd23769-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.529786 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.529611 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zhf\" (UniqueName: \"kubernetes.io/projected/b9181f5b-4c52-49ca-bb5d-a283bdd23769-kube-api-access-h5zhf\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.532198 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.532172 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9181f5b-4c52-49ca-bb5d-a283bdd23769-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.540207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.540161 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zhf\" (UniqueName: \"kubernetes.io/projected/b9181f5b-4c52-49ca-bb5d-a283bdd23769-kube-api-access-h5zhf\") pod \"monitoring-plugin-d5dcff7cf-8b26m\" (UID: \"b9181f5b-4c52-49ca-bb5d-a283bdd23769\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.599390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.599354 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"c94f6d4b7e9db3b4f17a81e9742e7b297c5246a1dc4abf5007f408c1f8ec58ab"} Jul 07 04:25:26.599559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.599442 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"a2309ae2af7c97509c782fe8d314174379934f6b592cd8d8120e748e00607db2"} Jul 07 04:25:26.599559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.599479 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"c9267072fad5b1043523232907295d0b2c957a4ffe5744f819655113d5b9b689"} Jul 07 04:25:26.617812 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.617779 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:26.732250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.732177 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82"] Jul 07 04:25:26.735409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.735389 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.739164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.739135 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 04:25:26.739289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.739173 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 04:25:26.740148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.740125 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 04:25:26.740266 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.740197 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 04:25:26.740467 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.740449 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vvfcb\"" Jul 07 04:25:26.740561 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.740498 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 04:25:26.747325 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.747306 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 04:25:26.755707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.755678 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82"] Jul 07 04:25:26.832682 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832646 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.832890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dzm\" (UniqueName: \"kubernetes.io/projected/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-kube-api-access-z5dzm\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.832890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832729 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.832890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832751 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.832890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.833088 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832919 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-serving-certs-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.833088 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-federate-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.833088 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.832996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-metrics-client-ca\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933585 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933552 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933585 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933586 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933787 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933609 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933787 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-serving-certs-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933923 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-federate-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.933980 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.933935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-metrics-client-ca\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.934059 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.934033 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.934121 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.934064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dzm\" (UniqueName: \"kubernetes.io/projected/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-kube-api-access-z5dzm\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.934421 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.934355 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-serving-certs-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.934562 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.934538 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.934628 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.934562 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-metrics-client-ca\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.936530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.936490 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.936644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.936530 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-federate-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.936644 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.936571 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-secret-telemeter-client\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.936923 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.936908 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-telemeter-client-tls\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:26.944415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:26.944391 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dzm\" (UniqueName: \"kubernetes.io/projected/5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd-kube-api-access-z5dzm\") pod \"telemeter-client-65bd4dbd5c-5wd82\" (UID: \"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd\") " pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:27.046153 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.046073 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" Jul 07 04:25:27.171048 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.170965 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:27.262301 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.262271 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m"] Jul 07 04:25:27.265589 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.265477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:27.267135 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:27.266692 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9181f5b_4c52_49ca_bb5d_a283bdd23769.slice/crio-1bd10eb7771c966b9e2a6deabd3de009d91283a341424a16f9300a1fe0bc9fe5 WatchSource:0}: Error finding container 1bd10eb7771c966b9e2a6deabd3de009d91283a341424a16f9300a1fe0bc9fe5: Status 404 returned error can't find the container with id 1bd10eb7771c966b9e2a6deabd3de009d91283a341424a16f9300a1fe0bc9fe5 Jul 07 04:25:27.288832 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.288753 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82"] Jul 07 04:25:27.292287 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:27.292256 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0f08c6_b2de_4e25_9e3d_1f6176ef4ffd.slice/crio-dd89e0ac419604dc5854c7840c4835dd3461268866dc9ee1b3cf50ddb2e722aa WatchSource:0}: Error finding container dd89e0ac419604dc5854c7840c4835dd3461268866dc9ee1b3cf50ddb2e722aa: Status 404 returned error can't find the container with id dd89e0ac419604dc5854c7840c4835dd3461268866dc9ee1b3cf50ddb2e722aa Jul 07 04:25:27.604367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.604329 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" event={"ID":"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd","Type":"ContainerStarted","Data":"dd89e0ac419604dc5854c7840c4835dd3461268866dc9ee1b3cf50ddb2e722aa"} Jul 07 04:25:27.607020 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.606970 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"014fd15187be8216f4ef5ea1a48919a43672a5d2f415e3def244617878d6bc2b"} Jul 07 04:25:27.607148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.607049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"a3993a1897ab4309d37f10370b154055cdbda2b32ad27333b0d8d4b510d3d7f3"} Jul 07 04:25:27.607148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.607082 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" event={"ID":"c1db23cf-71b6-4293-8837-21e139df2964","Type":"ContainerStarted","Data":"9d39cd3eb68c9f6d02a99deeb674a270d8c85673e7127f40fb6023b7e2026994"} Jul 07 04:25:27.607148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.607141 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:27.608061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.608042 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" event={"ID":"b9181f5b-4c52-49ca-bb5d-a283bdd23769","Type":"ContainerStarted","Data":"1bd10eb7771c966b9e2a6deabd3de009d91283a341424a16f9300a1fe0bc9fe5"} Jul 07 04:25:27.611155 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611131 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43"} Jul 07 04:25:27.611262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233"} Jul 07 04:25:27.611262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611229 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3"} Jul 07 04:25:27.611338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611269 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe"} Jul 07 04:25:27.611338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611296 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091"} Jul 07 04:25:27.611338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.611321 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerStarted","Data":"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c"} Jul 07 04:25:27.612140 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.612120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:27.631767 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.631711 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" podStartSLOduration=1.65576945 podStartE2EDuration="4.63169482s" totalImagesPullingTime="2.97592537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:27.6302475 +0000 UTC m=+85.297665692" watchObservedRunningTime="2026-07-07 04:25:27.63169482 +0000 UTC m=+85.299113012" Jul 07 04:25:27.664839 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.664788 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.919351698 podStartE2EDuration="5.664770603s" totalImagesPullingTime="2.726668514s" totalInitContainerRuntime="18.750391ms" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:27.663400722 +0000 UTC m=+85.330818915" watchObservedRunningTime="2026-07-07 04:25:27.664770603 +0000 UTC m=+85.332188795" Jul 07 04:25:27.713307 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.713269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:27.783398 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:27.783343 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:28.166438 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.166400 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:28.227246 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.227212 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:28.230903 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.230875 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.241050 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.241022 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 04:25:28.248698 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.248669 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:28.346995 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.346944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347180 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347032 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n457r\" (UniqueName: \"kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347180 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347121 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347180 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347344 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347344 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347291 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.347344 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.347321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.372438 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.372398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:28.448314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448273 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n457r\" (UniqueName: \"kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448354 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.448595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.448387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.449390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.449158 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.449390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.449204 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.449390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.449204 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.449586 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.449418 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.450926 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.450891 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.450926 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.450891 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.457314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.457289 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n457r\" (UniqueName: \"kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r\") pod \"console-7c47cff68d-m6n2m\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.481812 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.481780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:28.543663 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.543642 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:28.620783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.620693 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" event={"ID":"b9181f5b-4c52-49ca-bb5d-a283bdd23769","Type":"ContainerStarted","Data":"da00d5876245612f9435002cc397bc2d658617fd59b6f2fd5ca2ac923620b125"} Jul 07 04:25:28.622304 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.622278 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:28.627193 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.627146 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" Jul 07 04:25:28.652280 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.652216 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-8b26m" podStartSLOduration=1.4264016910000001 podStartE2EDuration="2.652193133s" totalImagesPullingTime="1.225791442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:28.650772 +0000 UTC m=+86.318190195" watchObservedRunningTime="2026-07-07 04:25:28.652193133 +0000 UTC m=+86.319611327" Jul 07 04:25:28.718394 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:28.718317 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:28.722157 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:28.722129 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d39f8d8_6e9f_487d_8270_4414a718be02.slice/crio-3de3bd3264487470aa81795a42901c1250e5e95e53a50aebdf4b1c28e66ef149 WatchSource:0}: Error finding container 3de3bd3264487470aa81795a42901c1250e5e95e53a50aebdf4b1c28e66ef149: Status 404 returned error can't find the container with id 3de3bd3264487470aa81795a42901c1250e5e95e53a50aebdf4b1c28e66ef149 Jul 07 04:25:29.625920 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.625813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47cff68d-m6n2m" event={"ID":"5d39f8d8-6e9f-487d-8270-4414a718be02","Type":"ContainerStarted","Data":"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da"} Jul 07 04:25:29.625920 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.625876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47cff68d-m6n2m" event={"ID":"5d39f8d8-6e9f-487d-8270-4414a718be02","Type":"ContainerStarted","Data":"3de3bd3264487470aa81795a42901c1250e5e95e53a50aebdf4b1c28e66ef149"} Jul 07 04:25:29.627984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.627959 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" event={"ID":"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd","Type":"ContainerStarted","Data":"1773f68a1b3255e4e9e99405f5d3a134b398f85ffbfe20ed88bca91d7b04ca63"} Jul 07 04:25:29.628110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.628018 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" event={"ID":"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd","Type":"ContainerStarted","Data":"55624ca3eefe875ee2da557ca17bffe749fc7aaf613bded7e1cfa66f7513043b"} Jul 07 04:25:29.628110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.628038 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" event={"ID":"5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd","Type":"ContainerStarted","Data":"7214bea80ffc1937421de6d81114fb4e69a797869f70e2b824c4fbb3273aec32"} Jul 07 04:25:29.664136 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.664084 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c47cff68d-m6n2m" podStartSLOduration=1.66406916 podStartE2EDuration="1.66406916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:29.663250823 +0000 UTC m=+87.330669016" watchObservedRunningTime="2026-07-07 04:25:29.66406916 +0000 UTC m=+87.331487352" Jul 07 04:25:29.678184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.678130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:29.701415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.701343 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65bd4dbd5c-5wd82" podStartSLOduration=1.628081886 podStartE2EDuration="3.701320527s" totalImagesPullingTime="2.073238641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:29.698193266 +0000 UTC m=+87.365611458" watchObservedRunningTime="2026-07-07 04:25:29.701320527 +0000 UTC m=+87.368738719" Jul 07 04:25:29.840229 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:29.840190 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:30.485706 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.485671 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:30.531085 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.531053 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:25:30.534429 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.534407 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.553600 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.553561 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:25:30.668205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zxv\" (UniqueName: \"kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668431 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.668802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.668492 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769537 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769440 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zxv\" (UniqueName: \"kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769537 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769501 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769764 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769764 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769877 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.769936 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.769881 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.770563 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.770536 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.770760 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.770734 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.770760 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.770658 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.770894 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.770810 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.771302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.771279 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.772247 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.772224 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.772445 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.772427 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.779598 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.779576 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zxv\" (UniqueName: \"kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv\") pod \"console-7bd9989fd-nbvxt\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.843645 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.843607 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:30.974509 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:30.974369 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:25:30.977195 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:25:30.977172 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a2a1d1_0f81_4f99_883c_1566b3242b5b.slice/crio-35902cd99b6e3d41b2b0a89155ea692d72b37b7cbe3b4746993ded49920360ca WatchSource:0}: Error finding container 35902cd99b6e3d41b2b0a89155ea692d72b37b7cbe3b4746993ded49920360ca: Status 404 returned error can't find the container with id 35902cd99b6e3d41b2b0a89155ea692d72b37b7cbe3b4746993ded49920360ca Jul 07 04:25:31.641216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:31.641177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd9989fd-nbvxt" event={"ID":"b3a2a1d1-0f81-4f99-883c-1566b3242b5b","Type":"ContainerStarted","Data":"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a"} Jul 07 04:25:31.641374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:31.641255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd9989fd-nbvxt" event={"ID":"b3a2a1d1-0f81-4f99-883c-1566b3242b5b","Type":"ContainerStarted","Data":"35902cd99b6e3d41b2b0a89155ea692d72b37b7cbe3b4746993ded49920360ca"} Jul 07 04:25:31.680713 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:31.680655 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd9989fd-nbvxt" podStartSLOduration=1.6806339750000001 podStartE2EDuration="1.680633975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:25:31.680063026 +0000 UTC m=+89.347481218" watchObservedRunningTime="2026-07-07 04:25:31.680633975 +0000 UTC m=+89.348052169" Jul 07 04:25:33.629068 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:33.629040 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-697c7f855-rc92g" Jul 07 04:25:34.093470 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:34.093435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:34.194329 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:34.194285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:38.544594 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:38.544553 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:40.844237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:40.844197 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:40.844678 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:40.844299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:40.849432 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:40.849410 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:41.683639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:41.683613 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:25:41.732756 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:41.732724 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:25:41.848638 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:41.848601 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:41.951957 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:41.951878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:42.058535 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:42.058505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:44.631563 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:44.631518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:44.709977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:44.709945 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:25:48.554891 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:48.554861 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zd488" Jul 07 04:25:56.660981 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:56.660921 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c47cff68d-m6n2m" podUID="5d39f8d8-6e9f-487d-8270-4414a718be02" containerName="console" containerID="cri-o://52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da" gracePeriod=15 Jul 07 04:25:56.905125 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:56.905102 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:57.002256 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002169 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002256 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002215 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002256 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002244 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002271 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002307 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002330 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n457r\" (UniqueName: \"kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002366 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config\") pod \"5d39f8d8-6e9f-487d-8270-4414a718be02\" (UID: \"5d39f8d8-6e9f-487d-8270-4414a718be02\") " Jul 07 04:25:57.002711 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002644 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:57.002843 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002811 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca" (OuterVolumeSpecName: "service-ca") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:57.002843 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002830 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:57.002938 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.002862 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config" (OuterVolumeSpecName: "console-config") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:25:57.004540 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.004515 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r" (OuterVolumeSpecName: "kube-api-access-n457r") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "kube-api-access-n457r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:25:57.004628 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.004539 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:25:57.004628 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.004590 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5d39f8d8-6e9f-487d-8270-4414a718be02" (UID: "5d39f8d8-6e9f-487d-8270-4414a718be02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:25:57.103492 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103441 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103492 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103486 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103492 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103501 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n457r\" (UniqueName: \"kubernetes.io/projected/5d39f8d8-6e9f-487d-8270-4414a718be02-kube-api-access-n457r\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103515 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103527 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103538 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5d39f8d8-6e9f-487d-8270-4414a718be02-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.103739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.103550 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5d39f8d8-6e9f-487d-8270-4414a718be02-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:25:57.740407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.740369 2560 generic.go:356] "Generic (PLEG): container finished" podID="5d39f8d8-6e9f-487d-8270-4414a718be02" containerID="52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da" exitCode=0 Jul 07 04:25:57.740407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.740409 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47cff68d-m6n2m" Jul 07 04:25:57.740869 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.740447 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47cff68d-m6n2m" event={"ID":"5d39f8d8-6e9f-487d-8270-4414a718be02","Type":"ContainerDied","Data":"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da"} Jul 07 04:25:57.740869 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.740496 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47cff68d-m6n2m" event={"ID":"5d39f8d8-6e9f-487d-8270-4414a718be02","Type":"ContainerDied","Data":"3de3bd3264487470aa81795a42901c1250e5e95e53a50aebdf4b1c28e66ef149"} Jul 07 04:25:57.740869 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.740515 2560 scope.go:117] "RemoveContainer" containerID="52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da" Jul 07 04:25:57.748407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.748389 2560 scope.go:117] "RemoveContainer" containerID="52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da" Jul 07 04:25:57.748682 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:25:57.748663 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da\": container with ID starting with 52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da not found: ID does not exist" containerID="52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da" Jul 07 04:25:57.748734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.748690 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da"} err="failed to get container status \"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da\": rpc error: code = NotFound desc = could not find container \"52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da\": container with ID starting with 52b4c04e17203fb8cca3d072662408b82caf05541337720ebef8b1f8d37391da not found: ID does not exist" Jul 07 04:25:57.767203 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.767180 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:57.784797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:57.784774 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c47cff68d-m6n2m"] Jul 07 04:25:59.235588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:25:59.235559 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d39f8d8-6e9f-487d-8270-4414a718be02" path="/var/lib/kubelet/pods/5d39f8d8-6e9f-487d-8270-4414a718be02/volumes" Jul 07 04:26:06.752836 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:06.752771 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-75bd8b9dc4-rdjlc" podUID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" containerName="console" containerID="cri-o://2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17" gracePeriod=15 Jul 07 04:26:06.995792 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:06.995770 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:26:07.088046 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.087989 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcwsw\" (UniqueName: \"kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088081 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088124 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088152 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088174 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088412 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088242 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert\") pod \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\" (UID: \"3fbf9c7e-348c-428d-a983-2b3a92769d5b\") " Jul 07 04:26:07.088538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088510 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:26:07.088604 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088513 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:26:07.088652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.088596 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config" (OuterVolumeSpecName: "console-config") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:26:07.090346 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.090319 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:07.090432 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.090359 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw" (OuterVolumeSpecName: "kube-api-access-fcwsw") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "kube-api-access-fcwsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:26:07.090432 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.090374 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3fbf9c7e-348c-428d-a983-2b3a92769d5b" (UID: "3fbf9c7e-348c-428d-a983-2b3a92769d5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:07.189472 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189442 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.189472 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189471 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.189667 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189484 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.189667 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189498 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.189667 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189509 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbf9c7e-348c-428d-a983-2b3a92769d5b-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.189667 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.189519 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwsw\" (UniqueName: \"kubernetes.io/projected/3fbf9c7e-348c-428d-a983-2b3a92769d5b-kube-api-access-fcwsw\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:07.779425 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.779387 2560 generic.go:356] "Generic (PLEG): container finished" podID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" containerID="2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17" exitCode=0 Jul 07 04:26:07.779832 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.779438 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bd8b9dc4-rdjlc" Jul 07 04:26:07.779832 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.779437 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd8b9dc4-rdjlc" event={"ID":"3fbf9c7e-348c-428d-a983-2b3a92769d5b","Type":"ContainerDied","Data":"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17"} Jul 07 04:26:07.779832 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.779488 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bd8b9dc4-rdjlc" event={"ID":"3fbf9c7e-348c-428d-a983-2b3a92769d5b","Type":"ContainerDied","Data":"724c27004982896a3018c97d0225d30e95e83b2664f270d1112f36e07a494649"} Jul 07 04:26:07.779832 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.779514 2560 scope.go:117] "RemoveContainer" containerID="2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17" Jul 07 04:26:07.787431 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.787416 2560 scope.go:117] "RemoveContainer" containerID="2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17" Jul 07 04:26:07.787660 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:07.787639 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17\": container with ID starting with 2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17 not found: ID does not exist" containerID="2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17" Jul 07 04:26:07.787709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.787668 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17"} err="failed to get container status \"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17\": rpc error: code = NotFound desc = could not find container \"2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17\": container with ID starting with 2ebb6d82e0d8f6dad6324f50146de2530268684ce76ae63927325db5a1c82c17 not found: ID does not exist" Jul 07 04:26:07.799305 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.799282 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:26:07.805297 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:07.805277 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75bd8b9dc4-rdjlc"] Jul 07 04:26:09.235264 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:09.235232 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" path="/var/lib/kubelet/pods/3fbf9c7e-348c-428d-a983-2b3a92769d5b/volumes" Jul 07 04:26:32.451674 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:32.451639 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:37.631385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:37.631353 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:37.688723 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:37.688691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:41.908635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.908601 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:41.909042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.908879 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="alertmanager" containerID="cri-o://98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" gracePeriod=120 Jul 07 04:26:41.909042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.908940 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-metric" containerID="cri-o://87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" gracePeriod=120 Jul 07 04:26:41.909042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.908988 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-web" containerID="cri-o://c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" gracePeriod=120 Jul 07 04:26:41.909237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.908999 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="prom-label-proxy" containerID="cri-o://3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" gracePeriod=120 Jul 07 04:26:41.909237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.909061 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="config-reloader" containerID="cri-o://e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" gracePeriod=120 Jul 07 04:26:41.909237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:41.909072 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy" containerID="cri-o://0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" gracePeriod=120 Jul 07 04:26:42.913661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913631 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" exitCode=0 Jul 07 04:26:42.913661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913656 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" exitCode=0 Jul 07 04:26:42.913661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913662 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" exitCode=0 Jul 07 04:26:42.913661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913668 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" exitCode=0 Jul 07 04:26:42.914211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913698 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43"} Jul 07 04:26:42.914211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913759 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3"} Jul 07 04:26:42.914211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091"} Jul 07 04:26:42.914211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:42.913810 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c"} Jul 07 04:26:43.255778 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.255759 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:43.275252 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275235 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275329 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275268 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275366 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275353 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275404 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275387 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275422 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275454 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275486 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275521 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275548 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275577 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275596 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275615 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.275855 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.275644 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99jg6\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6\") pod \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\" (UID: \"06ca606a-1106-4d1b-9682-a6f1dc684dd2\") " Jul 07 04:26:43.277051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.276558 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:26:43.277051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.276816 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:26:43.277051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.276993 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:26:43.278151 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.278114 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.278151 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.278143 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out" (OuterVolumeSpecName: "config-out") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:26:43.278360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.278327 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6" (OuterVolumeSpecName: "kube-api-access-99jg6") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "kube-api-access-99jg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:26:43.278360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.278342 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.278473 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.278411 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.279431 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.279403 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.279808 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.279773 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume" (OuterVolumeSpecName: "config-volume") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.279897 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.279857 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:26:43.283646 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.283573 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.290219 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.290192 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config" (OuterVolumeSpecName: "web-config") pod "06ca606a-1106-4d1b-9682-a6f1dc684dd2" (UID: "06ca606a-1106-4d1b-9682-a6f1dc684dd2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:26:43.376890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376837 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.376890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376876 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-main-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.376890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376887 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.376890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376896 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.376890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376905 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-volume\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376914 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-metrics-client-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376923 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-cluster-tls-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376932 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-tls-assets\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376940 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376948 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-alertmanager-main-db\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376956 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jg6\" (UniqueName: \"kubernetes.io/projected/06ca606a-1106-4d1b-9682-a6f1dc684dd2-kube-api-access-99jg6\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376964 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06ca606a-1106-4d1b-9682-a6f1dc684dd2-web-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.377184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.376971 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06ca606a-1106-4d1b-9682-a6f1dc684dd2-config-out\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:26:43.920803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920769 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" exitCode=0 Jul 07 04:26:43.920803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920794 2560 generic.go:356] "Generic (PLEG): container finished" podID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerID="c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" exitCode=0 Jul 07 04:26:43.921350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920848 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233"} Jul 07 04:26:43.921350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920859 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:43.921350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920904 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe"} Jul 07 04:26:43.921350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920932 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06ca606a-1106-4d1b-9682-a6f1dc684dd2","Type":"ContainerDied","Data":"d687b53e77376d410708ee6241defd79a2dbe2f02f4296d0c1368796e06861fa"} Jul 07 04:26:43.921621 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.920962 2560 scope.go:117] "RemoveContainer" containerID="3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" Jul 07 04:26:43.929898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.929884 2560 scope.go:117] "RemoveContainer" containerID="87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" Jul 07 04:26:43.936608 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.936592 2560 scope.go:117] "RemoveContainer" containerID="0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" Jul 07 04:26:43.942710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.942693 2560 scope.go:117] "RemoveContainer" containerID="c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" Jul 07 04:26:43.945611 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.945591 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:43.949994 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.949976 2560 scope.go:117] "RemoveContainer" containerID="e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" Jul 07 04:26:43.950374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.950355 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:43.956248 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.956233 2560 scope.go:117] "RemoveContainer" containerID="98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" Jul 07 04:26:43.962860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.962844 2560 scope.go:117] "RemoveContainer" containerID="7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621" Jul 07 04:26:43.969529 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.969512 2560 scope.go:117] "RemoveContainer" containerID="3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" Jul 07 04:26:43.969792 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.969772 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43\": container with ID starting with 3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43 not found: ID does not exist" containerID="3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" Jul 07 04:26:43.969873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.969809 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43"} err="failed to get container status \"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43\": rpc error: code = NotFound desc = could not find container \"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43\": container with ID starting with 3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43 not found: ID does not exist" Jul 07 04:26:43.969873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.969829 2560 scope.go:117] "RemoveContainer" containerID="87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" Jul 07 04:26:43.970098 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.970080 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233\": container with ID starting with 87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233 not found: ID does not exist" containerID="87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" Jul 07 04:26:43.970171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970107 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233"} err="failed to get container status \"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233\": rpc error: code = NotFound desc = could not find container \"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233\": container with ID starting with 87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233 not found: ID does not exist" Jul 07 04:26:43.970171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970129 2560 scope.go:117] "RemoveContainer" containerID="0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" Jul 07 04:26:43.970344 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.970327 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3\": container with ID starting with 0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3 not found: ID does not exist" containerID="0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" Jul 07 04:26:43.970408 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970353 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3"} err="failed to get container status \"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3\": rpc error: code = NotFound desc = could not find container \"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3\": container with ID starting with 0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3 not found: ID does not exist" Jul 07 04:26:43.970408 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970375 2560 scope.go:117] "RemoveContainer" containerID="c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" Jul 07 04:26:43.970607 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.970590 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe\": container with ID starting with c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe not found: ID does not exist" containerID="c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" Jul 07 04:26:43.970645 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970612 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe"} err="failed to get container status \"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe\": rpc error: code = NotFound desc = could not find container \"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe\": container with ID starting with c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe not found: ID does not exist" Jul 07 04:26:43.970645 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970640 2560 scope.go:117] "RemoveContainer" containerID="e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" Jul 07 04:26:43.970845 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.970831 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091\": container with ID starting with e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091 not found: ID does not exist" containerID="e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" Jul 07 04:26:43.970885 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970848 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091"} err="failed to get container status \"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091\": rpc error: code = NotFound desc = could not find container \"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091\": container with ID starting with e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091 not found: ID does not exist" Jul 07 04:26:43.970885 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.970860 2560 scope.go:117] "RemoveContainer" containerID="98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" Jul 07 04:26:43.971108 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.971091 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c\": container with ID starting with 98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c not found: ID does not exist" containerID="98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" Jul 07 04:26:43.971179 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971114 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c"} err="failed to get container status \"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c\": rpc error: code = NotFound desc = could not find container \"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c\": container with ID starting with 98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c not found: ID does not exist" Jul 07 04:26:43.971179 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971135 2560 scope.go:117] "RemoveContainer" containerID="7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621" Jul 07 04:26:43.971371 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:26:43.971355 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621\": container with ID starting with 7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621 not found: ID does not exist" containerID="7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621" Jul 07 04:26:43.971407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971376 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621"} err="failed to get container status \"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621\": rpc error: code = NotFound desc = could not find container \"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621\": container with ID starting with 7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621 not found: ID does not exist" Jul 07 04:26:43.971407 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971396 2560 scope.go:117] "RemoveContainer" containerID="3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43" Jul 07 04:26:43.971584 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971568 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43"} err="failed to get container status \"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43\": rpc error: code = NotFound desc = could not find container \"3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43\": container with ID starting with 3389bc6c8460339cb39cc12651712e4cdfda818c9a5c09bef58169d550c63b43 not found: ID does not exist" Jul 07 04:26:43.971651 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971586 2560 scope.go:117] "RemoveContainer" containerID="87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233" Jul 07 04:26:43.971813 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971793 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233"} err="failed to get container status \"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233\": rpc error: code = NotFound desc = could not find container \"87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233\": container with ID starting with 87413cf8916e4ef79a7283e2cdb86f948dc97a5713a5b690d04de35c914ec233 not found: ID does not exist" Jul 07 04:26:43.971865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.971814 2560 scope.go:117] "RemoveContainer" containerID="0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3" Jul 07 04:26:43.972064 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972046 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3"} err="failed to get container status \"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3\": rpc error: code = NotFound desc = could not find container \"0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3\": container with ID starting with 0558c71f67b853841eb21fa0d7f46fdf02e4004eacc9336dedccd2c8286c34f3 not found: ID does not exist" Jul 07 04:26:43.972131 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972066 2560 scope.go:117] "RemoveContainer" containerID="c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe" Jul 07 04:26:43.972298 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972282 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe"} err="failed to get container status \"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe\": rpc error: code = NotFound desc = could not find container \"c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe\": container with ID starting with c517bdf350f49ca4302d65cf6896496f1b7c28a376c1a652aa949dfe5e7209fe not found: ID does not exist" Jul 07 04:26:43.972351 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972298 2560 scope.go:117] "RemoveContainer" containerID="e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091" Jul 07 04:26:43.972486 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972470 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091"} err="failed to get container status \"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091\": rpc error: code = NotFound desc = could not find container \"e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091\": container with ID starting with e961a60781c9ed714370371dd3c05b064a52fe5c6559df117cb59b55ca530091 not found: ID does not exist" Jul 07 04:26:43.972529 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972486 2560 scope.go:117] "RemoveContainer" containerID="98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c" Jul 07 04:26:43.972662 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972648 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c"} err="failed to get container status \"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c\": rpc error: code = NotFound desc = could not find container \"98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c\": container with ID starting with 98e42b11ff4f83a46e07929723f0e427dcf916376642833fb0e8b4ef5c7bf72c not found: ID does not exist" Jul 07 04:26:43.972662 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972661 2560 scope.go:117] "RemoveContainer" containerID="7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621" Jul 07 04:26:43.972866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.972847 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621"} err="failed to get container status \"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621\": rpc error: code = NotFound desc = could not find container \"7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621\": container with ID starting with 7427da0ca996391e7883131f6bd56f884966194282ad37d55a7fcf467be2a621 not found: ID does not exist" Jul 07 04:26:43.975106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975087 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:43.975389 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975376 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="config-reloader" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975391 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="config-reloader" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975398 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975403 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975414 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d39f8d8-6e9f-487d-8270-4414a718be02" containerName="console" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975422 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d39f8d8-6e9f-487d-8270-4414a718be02" containerName="console" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975433 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="alertmanager" Jul 07 04:26:43.975440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975441 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="alertmanager" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975452 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" containerName="console" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975458 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" containerName="console" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975464 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="prom-label-proxy" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975469 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="prom-label-proxy" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975477 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="init-config-reloader" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975482 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="init-config-reloader" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975488 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-metric" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975492 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-metric" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975503 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-web" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975508 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-web" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975550 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="config-reloader" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975558 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d39f8d8-6e9f-487d-8270-4414a718be02" containerName="console" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975564 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-web" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975569 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbf9c7e-348c-428d-a983-2b3a92769d5b" containerName="console" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975574 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975578 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="kube-rbac-proxy-metric" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975583 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="prom-label-proxy" Jul 07 04:26:43.975658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.975591 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" containerName="alertmanager" Jul 07 04:26:43.978673 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.978658 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:43.983997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.983970 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 04:26:43.983997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.983993 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 04:26:43.984191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984020 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 04:26:43.984191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984060 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-njbx4\"" Jul 07 04:26:43.984302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984198 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 04:26:43.984909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984889 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 04:26:43.985029 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984942 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 04:26:43.985029 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.984999 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 04:26:43.985029 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.985024 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 04:26:43.988713 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.988694 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 04:26:43.993100 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:43.992953 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:44.084689 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084655 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.084848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-config-out\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.084848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbppl\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-kube-api-access-bbppl\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.084848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084800 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-web-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.084848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084871 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084893 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-config-volume\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084908 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084940 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.085051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.084998 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.185997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-config-volume\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186080 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186131 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186197 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186272 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-config-out\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186297 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbppl\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-kube-api-access-bbppl\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186355 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-web-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186779 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186416 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.186779 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186675 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.187128 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.186916 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.187363 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.187341 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e861919e-86ed-4b64-a450-b0500f152495-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189049 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e861919e-86ed-4b64-a450-b0500f152495-config-out\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189308 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189328 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-config-volume\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189602 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189446 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189602 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189553 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189696 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189681 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.189852 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.189834 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.190051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.190031 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-web-config\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.191183 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.191160 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e861919e-86ed-4b64-a450-b0500f152495-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.197304 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.197286 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbppl\" (UniqueName: \"kubernetes.io/projected/e861919e-86ed-4b64-a450-b0500f152495-kube-api-access-bbppl\") pod \"alertmanager-main-0\" (UID: \"e861919e-86ed-4b64-a450-b0500f152495\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.315973 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.315927 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 04:26:44.440367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.440299 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:26:44.442995 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.442979 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.453303 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.453261 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 04:26:44.454979 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.454960 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:26:44.456585 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:26:44.456564 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode861919e_86ed_4b64_a450_b0500f152495.slice/crio-f7481c4050a80b5cc164fa9b6749fedd84d9486ea19458338e32d2b89d45754d WatchSource:0}: Error finding container f7481c4050a80b5cc164fa9b6749fedd84d9486ea19458338e32d2b89d45754d: Status 404 returned error can't find the container with id f7481c4050a80b5cc164fa9b6749fedd84d9486ea19458338e32d2b89d45754d Jul 07 04:26:44.488842 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.488820 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.488937 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.488863 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.488937 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.488922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.489019 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.488979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.489080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.489060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.489123 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.489097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.489163 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.489124 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs9h\" (UniqueName: \"kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.494452 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.494431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:44.589658 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589800 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589800 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589703 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589800 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589721 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589950 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589836 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs9h\" (UniqueName: \"kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589950 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.589950 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.589904 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.590476 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.590456 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.590601 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.590574 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.590686 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.590662 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.590731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.590683 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.592063 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.592044 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.592156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.592141 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.595107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.595086 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:44.597272 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.597257 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs9h\" (UniqueName: \"kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h\") pod \"console-75fc44b88d-wcv56\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.695917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.695838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:44.752538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.752507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:44.873415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.873395 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:26:44.875484 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:26:44.875459 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90bf5a8_b0de_4785_8186_84dad4ec9008.slice/crio-f9db41f3620ece59927ba89bed45b9a8e55314a2eb7ff8b7d1f205908c5c9be8 WatchSource:0}: Error finding container f9db41f3620ece59927ba89bed45b9a8e55314a2eb7ff8b7d1f205908c5c9be8: Status 404 returned error can't find the container with id f9db41f3620ece59927ba89bed45b9a8e55314a2eb7ff8b7d1f205908c5c9be8 Jul 07 04:26:44.929070 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.929040 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fc44b88d-wcv56" event={"ID":"e90bf5a8-b0de-4785-8186-84dad4ec9008","Type":"ContainerStarted","Data":"f9db41f3620ece59927ba89bed45b9a8e55314a2eb7ff8b7d1f205908c5c9be8"} Jul 07 04:26:44.930863 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.930845 2560 generic.go:356] "Generic (PLEG): container finished" podID="e861919e-86ed-4b64-a450-b0500f152495" containerID="04e437d7102b8c64b9f6fec8c62f80dcf57184c52ad492c4a5b1d88e421f6114" exitCode=0 Jul 07 04:26:44.930942 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.930922 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerDied","Data":"04e437d7102b8c64b9f6fec8c62f80dcf57184c52ad492c4a5b1d88e421f6114"} Jul 07 04:26:44.931024 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:44.930990 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"f7481c4050a80b5cc164fa9b6749fedd84d9486ea19458338e32d2b89d45754d"} Jul 07 04:26:45.238761 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.238733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:45.240581 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.240559 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ca606a-1106-4d1b-9682-a6f1dc684dd2" path="/var/lib/kubelet/pods/06ca606a-1106-4d1b-9682-a6f1dc684dd2/volumes" Jul 07 04:26:45.337164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.337129 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:45.936613 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.936579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fc44b88d-wcv56" event={"ID":"e90bf5a8-b0de-4785-8186-84dad4ec9008","Type":"ContainerStarted","Data":"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937"} Jul 07 04:26:45.942395 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"6ea6fdd357ea45ccc309376c137b791e3bc7884e98dc10618f6f20b62c35d0f9"} Jul 07 04:26:45.942523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942422 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"d5f5780bfb3d5d88bf923cc51be5a43315592934c81e25a949feb2736fe7fb5b"} Jul 07 04:26:45.942523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"aa86ee0e704170a8a47b6575be2a396dcb3b9401dead4ae5b8395fc25ee6f5cd"} Jul 07 04:26:45.942523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942474 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"187d283049af0f9e0fc8fb37e5d0eea5dea0a7c0b6cccdbba8dc88a9031cbb2e"} Jul 07 04:26:45.942523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"dd285f8b44d0a1d0383a92ee0e368f68cb7377a8673367746133eb7b50caf59c"} Jul 07 04:26:45.942649 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.942533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e861919e-86ed-4b64-a450-b0500f152495","Type":"ContainerStarted","Data":"de0e782daf04f06f75176674634574ff2386c32db5f4527f678944c211eff7a0"} Jul 07 04:26:45.959821 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.959333 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75fc44b88d-wcv56" podStartSLOduration=1.959314951 podStartE2EDuration="1.959314951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:26:45.958526718 +0000 UTC m=+163.625944928" watchObservedRunningTime="2026-07-07 04:26:45.959314951 +0000 UTC m=+163.626733144" Jul 07 04:26:45.990613 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:45.990539 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9767684770000002 podStartE2EDuration="2.990521671s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.753194ms" isStatefulPod=false podCreationTimestamp="2026-07-07 04:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:26:45.988603953 +0000 UTC m=+163.656022144" watchObservedRunningTime="2026-07-07 04:26:45.990521671 +0000 UTC m=+163.657939864" Jul 07 04:26:49.655669 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:49.655624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:49.710038 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:49.709991 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:54.753424 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:54.753336 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:54.753424 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:54.753403 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:54.758055 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:54.758034 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:54.983195 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:54.983165 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:26:55.033907 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:55.033818 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:26:55.139575 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:55.139543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:55.202685 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:55.202657 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:55.259466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:55.259435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:59.615481 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:59.615442 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:26:59.717272 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:26:59.717244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:02.043332 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:02.043304 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:02.138134 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:02.138093 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:03.635213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:03.635178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:18.392970 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:18.392934 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:20.053829 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.053788 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7bd9989fd-nbvxt" podUID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" containerName="console" containerID="cri-o://d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a" gracePeriod=15 Jul 07 04:27:20.292159 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.292137 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:27:20.294235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294219 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294253 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294277 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2zxv\" (UniqueName: \"kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294371 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294299 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294371 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294329 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294371 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294516 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294394 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config\") pod \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\" (UID: \"b3a2a1d1-0f81-4f99-883c-1566b3242b5b\") " Jul 07 04:27:20.294894 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294655 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:27:20.294894 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294784 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:27:20.295061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294915 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:27:20.295061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.294952 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config" (OuterVolumeSpecName: "console-config") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:27:20.296348 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.296328 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:27:20.296462 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.296373 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:27:20.296527 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.296517 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv" (OuterVolumeSpecName: "kube-api-access-n2zxv") pod "b3a2a1d1-0f81-4f99-883c-1566b3242b5b" (UID: "b3a2a1d1-0f81-4f99-883c-1566b3242b5b"). InnerVolumeSpecName "kube-api-access-n2zxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:27:20.395302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395274 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395298 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395307 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zxv\" (UniqueName: \"kubernetes.io/projected/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-kube-api-access-n2zxv\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395316 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395325 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395333 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:20.395525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:20.395341 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3a2a1d1-0f81-4f99-883c-1566b3242b5b-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:21.072321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.072271 2560 generic.go:356] "Generic (PLEG): container finished" podID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" containerID="d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a" exitCode=0 Jul 07 04:27:21.072701 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.072331 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd9989fd-nbvxt" Jul 07 04:27:21.072701 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.072361 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd9989fd-nbvxt" event={"ID":"b3a2a1d1-0f81-4f99-883c-1566b3242b5b","Type":"ContainerDied","Data":"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a"} Jul 07 04:27:21.072701 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.072441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd9989fd-nbvxt" event={"ID":"b3a2a1d1-0f81-4f99-883c-1566b3242b5b","Type":"ContainerDied","Data":"35902cd99b6e3d41b2b0a89155ea692d72b37b7cbe3b4746993ded49920360ca"} Jul 07 04:27:21.072701 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.072463 2560 scope.go:117] "RemoveContainer" containerID="d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a" Jul 07 04:27:21.080961 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.080945 2560 scope.go:117] "RemoveContainer" containerID="d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a" Jul 07 04:27:21.081214 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:27:21.081196 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a\": container with ID starting with d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a not found: ID does not exist" containerID="d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a" Jul 07 04:27:21.081281 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.081227 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a"} err="failed to get container status \"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a\": rpc error: code = NotFound desc = could not find container \"d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a\": container with ID starting with d2c60cab71722692837ba8a5088bc835bb4f9807fc57b06c0f2f2218f6ee360a not found: ID does not exist" Jul 07 04:27:21.092853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.092831 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:27:21.096796 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.096771 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd9989fd-nbvxt"] Jul 07 04:27:21.234606 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:21.234557 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" path="/var/lib/kubelet/pods/b3a2a1d1-0f81-4f99-883c-1566b3242b5b/volumes" Jul 07 04:27:36.571939 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:36.571888 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:36.671808 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:36.671752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:37.249338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.249302 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4"] Jul 07 04:27:37.249584 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.249573 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" containerName="console" Jul 07 04:27:37.249627 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.249586 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" containerName="console" Jul 07 04:27:37.249660 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.249645 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a2a1d1-0f81-4f99-883c-1566b3242b5b" containerName="console" Jul 07 04:27:37.252515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.252499 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.255386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.255357 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m2hxp\"" Jul 07 04:27:37.255386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.255380 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 04:27:37.255584 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.255360 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 04:27:37.262893 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.262867 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4"] Jul 07 04:27:37.308174 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.308130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.308349 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.308205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.308349 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.308258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjvms\" (UniqueName: \"kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.408788 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.408753 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.408970 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.408821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjvms\" (UniqueName: \"kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.408970 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.408840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.409196 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.409181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.409235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.409204 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.418373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.418350 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjvms\" (UniqueName: \"kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.561370 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.561271 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:37.714423 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:37.714394 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4"] Jul 07 04:27:37.717531 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:27:37.717502 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7e14ae_3ef3_4ed9_a602_27bed6a734d5.slice/crio-bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a WatchSource:0}: Error finding container bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a: Status 404 returned error can't find the container with id bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a Jul 07 04:27:38.141835 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:38.141797 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerStarted","Data":"bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a"} Jul 07 04:27:43.757935 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.757853 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j"] Jul 07 04:27:43.762070 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.762025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:43.762389 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.762373 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.765058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.765039 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 04:27:43.766238 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.766217 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 04:27:43.766382 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.766358 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 04:27:43.766382 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.766217 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 04:27:43.766834 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.766664 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6rjfd\"" Jul 07 04:27:43.772026 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.769589 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j"] Jul 07 04:27:43.833491 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.833458 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:43.840398 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.840371 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6"] Jul 07 04:27:43.843460 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.843445 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.846719 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.846700 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 04:27:43.852052 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.852030 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6"] Jul 07 04:27:43.856466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.856444 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35b1e498-faf3-4d76-a33b-94abc5c701a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.856580 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.856499 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlhq\" (UniqueName: \"kubernetes.io/projected/35b1e498-faf3-4d76-a33b-94abc5c701a5-kube-api-access-kmlhq\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.856580 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.856519 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9113d2c1-2404-491a-ae31-5a656dfa7cbd-klusterlet-config\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.856699 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.856582 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7psd\" (UniqueName: \"kubernetes.io/projected/9113d2c1-2404-491a-ae31-5a656dfa7cbd-kube-api-access-l7psd\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.856699 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.856632 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9113d2c1-2404-491a-ae31-5a656dfa7cbd-tmp\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.902069 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.902045 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:43.958045 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.957981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlhq\" (UniqueName: \"kubernetes.io/projected/35b1e498-faf3-4d76-a33b-94abc5c701a5-kube-api-access-kmlhq\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.958230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.958065 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9113d2c1-2404-491a-ae31-5a656dfa7cbd-klusterlet-config\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.958230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.958100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7psd\" (UniqueName: \"kubernetes.io/projected/9113d2c1-2404-491a-ae31-5a656dfa7cbd-kube-api-access-l7psd\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.958230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.958148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9113d2c1-2404-491a-ae31-5a656dfa7cbd-tmp\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.958230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.958200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35b1e498-faf3-4d76-a33b-94abc5c701a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.958657 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.958634 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9113d2c1-2404-491a-ae31-5a656dfa7cbd-tmp\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.960750 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.960729 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9113d2c1-2404-491a-ae31-5a656dfa7cbd-klusterlet-config\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:43.960926 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.960900 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35b1e498-faf3-4d76-a33b-94abc5c701a5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.967199 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.967125 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlhq\" (UniqueName: \"kubernetes.io/projected/35b1e498-faf3-4d76-a33b-94abc5c701a5-kube-api-access-kmlhq\") pod \"managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j\" (UID: \"35b1e498-faf3-4d76-a33b-94abc5c701a5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:43.967371 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:43.967238 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7psd\" (UniqueName: \"kubernetes.io/projected/9113d2c1-2404-491a-ae31-5a656dfa7cbd-kube-api-access-l7psd\") pod \"klusterlet-addon-workmgr-857cdd8b79-644d6\" (UID: \"9113d2c1-2404-491a-ae31-5a656dfa7cbd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:44.086115 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.086082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" Jul 07 04:27:44.153165 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.153139 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:44.171747 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.171221 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerID="d2fdc5a4eb8258f022b55858ee70e61ac8ee0d228e3780a5b9514103271b02fb" exitCode=0 Jul 07 04:27:44.171747 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.171278 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerDied","Data":"d2fdc5a4eb8258f022b55858ee70e61ac8ee0d228e3780a5b9514103271b02fb"} Jul 07 04:27:44.214744 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.214699 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j"] Jul 07 04:27:44.218013 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:27:44.217970 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35b1e498_faf3_4d76_a33b_94abc5c701a5.slice/crio-3698bdf89a9ecab46007e2e722720c10743ca7065ad7ca95463f7424e911732f WatchSource:0}: Error finding container 3698bdf89a9ecab46007e2e722720c10743ca7065ad7ca95463f7424e911732f: Status 404 returned error can't find the container with id 3698bdf89a9ecab46007e2e722720c10743ca7065ad7ca95463f7424e911732f Jul 07 04:27:44.284017 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.283971 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6"] Jul 07 04:27:44.286917 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:27:44.286888 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9113d2c1_2404_491a_ae31_5a656dfa7cbd.slice/crio-4fb2f20f62f1c996a9672fc26a94a9daf7d2e6df9e859d740320bc87f4ef4580 WatchSource:0}: Error finding container 4fb2f20f62f1c996a9672fc26a94a9daf7d2e6df9e859d740320bc87f4ef4580: Status 404 returned error can't find the container with id 4fb2f20f62f1c996a9672fc26a94a9daf7d2e6df9e859d740320bc87f4ef4580 Jul 07 04:27:44.973281 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:44.973242 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:45.180378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:45.180340 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" event={"ID":"9113d2c1-2404-491a-ae31-5a656dfa7cbd","Type":"ContainerStarted","Data":"4fb2f20f62f1c996a9672fc26a94a9daf7d2e6df9e859d740320bc87f4ef4580"} Jul 07 04:27:45.181927 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:45.181893 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" event={"ID":"35b1e498-faf3-4d76-a33b-94abc5c701a5","Type":"ContainerStarted","Data":"3698bdf89a9ecab46007e2e722720c10743ca7065ad7ca95463f7424e911732f"} Jul 07 04:27:49.200498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.200466 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerID="dee814babb657324c7d7708ce361be7cdb5b9e046b14133abb49b52bffa23d4a" exitCode=0 Jul 07 04:27:49.200907 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.200554 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerDied","Data":"dee814babb657324c7d7708ce361be7cdb5b9e046b14133abb49b52bffa23d4a"} Jul 07 04:27:49.202197 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.202172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" event={"ID":"9113d2c1-2404-491a-ae31-5a656dfa7cbd","Type":"ContainerStarted","Data":"74a4c940a4a2c92422c8968857eb3ff639cc53c4b2d366055b5b4575b49b19b8"} Jul 07 04:27:49.202275 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.202227 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:49.203711 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.203684 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" event={"ID":"35b1e498-faf3-4d76-a33b-94abc5c701a5","Type":"ContainerStarted","Data":"f85ace7049217a0c4484f6b2bf1df9b92ed4e0bd685f673e43a47d0eb10d1366"} Jul 07 04:27:49.204358 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.204324 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" Jul 07 04:27:49.240741 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.240690 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668dbcbcf6-ncs9j" podStartSLOduration=1.697817105 podStartE2EDuration="6.240675525s" totalImagesPullingTime="4.54285842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:27:49.239307623 +0000 UTC m=+226.906725817" watchObservedRunningTime="2026-07-07 04:27:49.240675525 +0000 UTC m=+226.908093776" Jul 07 04:27:49.258538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:49.258495 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-857cdd8b79-644d6" podStartSLOduration=1.767997089 podStartE2EDuration="6.258481691s" totalImagesPullingTime="4.490484602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:27:49.257709285 +0000 UTC m=+226.925127479" watchObservedRunningTime="2026-07-07 04:27:49.258481691 +0000 UTC m=+226.925899884" Jul 07 04:27:54.062519 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:54.062485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:54.114934 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:54.114896 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:27:57.238649 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:57.238618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerStarted","Data":"bf6da4087d456af7e0147ae00f3399cdb97e728ef0c488ea73b299d5bf037134"} Jul 07 04:27:57.259311 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:57.259255 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" podStartSLOduration=1.573093962 podStartE2EDuration="20.259240324s" totalImagesPullingTime="18.686146362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:27:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:27:57.257355202 +0000 UTC m=+234.924773394" watchObservedRunningTime="2026-07-07 04:27:57.259240324 +0000 UTC m=+234.926658515" Jul 07 04:27:58.244656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:58.244625 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerID="bf6da4087d456af7e0147ae00f3399cdb97e728ef0c488ea73b299d5bf037134" exitCode=0 Jul 07 04:27:58.245071 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:58.244675 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerDied","Data":"bf6da4087d456af7e0147ae00f3399cdb97e728ef0c488ea73b299d5bf037134"} Jul 07 04:27:59.427597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.427565 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:27:59.499258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.499157 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle\") pod \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " Jul 07 04:27:59.499258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.499221 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjvms\" (UniqueName: \"kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms\") pod \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " Jul 07 04:27:59.499491 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.499295 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util\") pod \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\" (UID: \"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5\") " Jul 07 04:27:59.499966 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.499933 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle" (OuterVolumeSpecName: "bundle") pod "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" (UID: "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:27:59.501511 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.501489 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms" (OuterVolumeSpecName: "kube-api-access-hjvms") pod "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" (UID: "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5"). InnerVolumeSpecName "kube-api-access-hjvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:27:59.503669 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.503648 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util" (OuterVolumeSpecName: "util") pod "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" (UID: "8d7e14ae-3ef3-4ed9-a602-27bed6a734d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:27:59.604906 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.600862 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjvms\" (UniqueName: \"kubernetes.io/projected/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-kube-api-access-hjvms\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:59.604906 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.600916 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:27:59.604906 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:27:59.600931 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d7e14ae-3ef3-4ed9-a602-27bed6a734d5-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:00.253849 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:00.253813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" event={"ID":"8d7e14ae-3ef3-4ed9-a602-27bed6a734d5","Type":"ContainerDied","Data":"bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a"} Jul 07 04:28:00.253849 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:00.253838 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zjpf4" Jul 07 04:28:00.254114 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:00.253860 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf6675f828779923bb1418b6b6f281f9abc5d14b0ae305cd47e800611b0cee2a" Jul 07 04:28:03.339687 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:03.339654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:04.264998 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:04.264961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:06.950320 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:06.950286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:07.051844 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:07.051800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:08.876308 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:08.876260 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:08.972069 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:08.972034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:09.163635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.163535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:09.285372 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.285345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:09.879860 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.879827 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8"] Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880112 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="util" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880123 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="util" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880140 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="extract" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880155 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="extract" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880166 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="pull" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880171 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="pull" Jul 07 04:28:09.880263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.880214 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d7e14ae-3ef3-4ed9-a602-27bed6a734d5" containerName="extract" Jul 07 04:28:09.883573 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.883555 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:09.886570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.886549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m2hxp\"" Jul 07 04:28:09.886744 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.886727 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 04:28:09.887753 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.887739 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 04:28:09.892590 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.892570 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8"] Jul 07 04:28:09.997153 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.997118 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:09.997153 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.997164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr569\" (UniqueName: \"kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:09.997390 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:09.997203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.097666 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.097635 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.097666 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.097676 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr569\" (UniqueName: \"kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.097912 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.097716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.098052 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.098033 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.098122 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.098066 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.106484 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.106459 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr569\" (UniqueName: \"kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.193654 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.193568 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:10.349623 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.349600 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8"] Jul 07 04:28:10.351960 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:28:10.351928 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609be7a5_7ee6_4d42_a124_f6259c8f5960.slice/crio-b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9 WatchSource:0}: Error finding container b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9: Status 404 returned error can't find the container with id b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9 Jul 07 04:28:10.883409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:10.883373 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:11.294697 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:11.294596 2560 generic.go:356] "Generic (PLEG): container finished" podID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerID="17dedf48539a75e239192449687043e655f216c689a3a295713a4901717ea9d5" exitCode=0 Jul 07 04:28:11.294697 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:11.294666 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerDied","Data":"17dedf48539a75e239192449687043e655f216c689a3a295713a4901717ea9d5"} Jul 07 04:28:11.294989 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:11.294709 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerStarted","Data":"b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9"} Jul 07 04:28:11.294989 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:11.294833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:13.307160 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:13.307117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerStarted","Data":"b802fcb709780b3cbaa2dd8491247401878b28c64ad3ed9e739843b7f5f85dbf"} Jul 07 04:28:13.957141 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:13.957107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:14.018374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:14.018336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:14.313374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:14.313283 2560 generic.go:356] "Generic (PLEG): container finished" podID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerID="b802fcb709780b3cbaa2dd8491247401878b28c64ad3ed9e739843b7f5f85dbf" exitCode=0 Jul 07 04:28:14.313374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:14.313352 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerDied","Data":"b802fcb709780b3cbaa2dd8491247401878b28c64ad3ed9e739843b7f5f85dbf"} Jul 07 04:28:15.318594 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:15.318561 2560 generic.go:356] "Generic (PLEG): container finished" podID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerID="102f968d431a30a674cf23513bc919bed8dfc0c6b9d0d16d1b48c57e972bb436" exitCode=0 Jul 07 04:28:15.319069 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:15.318619 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerDied","Data":"102f968d431a30a674cf23513bc919bed8dfc0c6b9d0d16d1b48c57e972bb436"} Jul 07 04:28:16.448688 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.448661 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:16.455569 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.455530 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jr569\" (UniqueName: \"kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569\") pod \"609be7a5-7ee6-4d42-a124-f6259c8f5960\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " Jul 07 04:28:16.455734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.455582 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util\") pod \"609be7a5-7ee6-4d42-a124-f6259c8f5960\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " Jul 07 04:28:16.455734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.455617 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle\") pod \"609be7a5-7ee6-4d42-a124-f6259c8f5960\" (UID: \"609be7a5-7ee6-4d42-a124-f6259c8f5960\") " Jul 07 04:28:16.456063 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.456036 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle" (OuterVolumeSpecName: "bundle") pod "609be7a5-7ee6-4d42-a124-f6259c8f5960" (UID: "609be7a5-7ee6-4d42-a124-f6259c8f5960"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:28:16.457569 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.457542 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569" (OuterVolumeSpecName: "kube-api-access-jr569") pod "609be7a5-7ee6-4d42-a124-f6259c8f5960" (UID: "609be7a5-7ee6-4d42-a124-f6259c8f5960"). InnerVolumeSpecName "kube-api-access-jr569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:28:16.462705 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.462680 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util" (OuterVolumeSpecName: "util") pod "609be7a5-7ee6-4d42-a124-f6259c8f5960" (UID: "609be7a5-7ee6-4d42-a124-f6259c8f5960"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:28:16.556412 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.556377 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr569\" (UniqueName: \"kubernetes.io/projected/609be7a5-7ee6-4d42-a124-f6259c8f5960-kube-api-access-jr569\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:16.556412 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.556405 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:16.556412 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.556417 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/609be7a5-7ee6-4d42-a124-f6259c8f5960-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:16.926053 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:16.925993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:17.327913 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:17.327881 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" event={"ID":"609be7a5-7ee6-4d42-a124-f6259c8f5960","Type":"ContainerDied","Data":"b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9"} Jul 07 04:28:17.328092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:17.327929 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17b6cae81b705388ff8cf0d16ae4040adb784581218d9a4b5b53cf88c1a1ab9" Jul 07 04:28:17.328092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:17.327927 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7q5v8" Jul 07 04:28:19.681041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:19.680994 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:22.758201 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:22.758166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:22.856917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:22.856887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:24.639857 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.639773 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt"] Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640083 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="util" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640094 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="util" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640109 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="pull" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640115 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="pull" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640122 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="extract" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640128 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="extract" Jul 07 04:28:24.640271 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.640187 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="609be7a5-7ee6-4d42-a124-f6259c8f5960" containerName="extract" Jul 07 04:28:24.642899 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.642883 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.645780 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.645757 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 04:28:24.645907 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.645835 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-bqchv\"" Jul 07 04:28:24.645907 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.645883 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 04:28:24.654024 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.653984 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt"] Jul 07 04:28:24.715762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.715724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ncg\" (UniqueName: \"kubernetes.io/projected/cd42f678-7454-4bd1-a0ab-460094410392-kube-api-access-w5ncg\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.715762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.715760 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd42f678-7454-4bd1-a0ab-460094410392-tmp\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.816830 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.816788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ncg\" (UniqueName: \"kubernetes.io/projected/cd42f678-7454-4bd1-a0ab-460094410392-kube-api-access-w5ncg\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.816830 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.816832 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd42f678-7454-4bd1-a0ab-460094410392-tmp\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.817211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.817194 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd42f678-7454-4bd1-a0ab-460094410392-tmp\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.824917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.824894 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ncg\" (UniqueName: \"kubernetes.io/projected/cd42f678-7454-4bd1-a0ab-460094410392-kube-api-access-w5ncg\") pod \"openshift-lws-operator-fd8ccff4c-56rgt\" (UID: \"cd42f678-7454-4bd1-a0ab-460094410392\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:24.952039 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:24.951929 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" Jul 07 04:28:25.080341 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:25.080316 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt"] Jul 07 04:28:25.082972 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:28:25.082931 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd42f678_7454_4bd1_a0ab_460094410392.slice/crio-1b2024c690d443d2e21a180f48156433e9f99f067d658363abba94a5fb20e2bf WatchSource:0}: Error finding container 1b2024c690d443d2e21a180f48156433e9f99f067d658363abba94a5fb20e2bf: Status 404 returned error can't find the container with id 1b2024c690d443d2e21a180f48156433e9f99f067d658363abba94a5fb20e2bf Jul 07 04:28:25.358360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:25.358328 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" event={"ID":"cd42f678-7454-4bd1-a0ab-460094410392","Type":"ContainerStarted","Data":"1b2024c690d443d2e21a180f48156433e9f99f067d658363abba94a5fb20e2bf"} Jul 07 04:28:27.367252 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:27.367214 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" event={"ID":"cd42f678-7454-4bd1-a0ab-460094410392","Type":"ContainerStarted","Data":"1e87099ef1ff090183f7318b19302a056cfde6db20a5d8e42923f498da0c1e9a"} Jul 07 04:28:27.383762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:27.383714 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-56rgt" podStartSLOduration=1.5390491389999998 podStartE2EDuration="3.383697891s" totalImagesPullingTime="1.844648752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:27.38297841 +0000 UTC m=+265.050396602" watchObservedRunningTime="2026-07-07 04:28:27.383697891 +0000 UTC m=+265.051116084" Jul 07 04:28:28.229158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:28.229120 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:28.329905 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:28.329862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:30.114175 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.114145 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:30.255990 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.255957 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d"] Jul 07 04:28:30.258265 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.258250 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.260812 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.260789 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-m2hxp\"" Jul 07 04:28:30.260812 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.260804 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 04:28:30.261882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.261864 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 04:28:30.267559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.267531 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d"] Jul 07 04:28:30.365523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.365446 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.365523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.365485 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrxd\" (UniqueName: \"kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.365523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.365504 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.466259 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.466227 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.466461 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.466277 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrxd\" (UniqueName: \"kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.466461 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.466297 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.466696 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.466673 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.466755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.466697 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.474888 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.474866 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrxd\" (UniqueName: \"kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.567962 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.567930 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:30.692576 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:30.692541 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d"] Jul 07 04:28:30.694580 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:28:30.694555 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472fd6bd_b968_488c_a923_45be7b2261a5.slice/crio-8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca WatchSource:0}: Error finding container 8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca: Status 404 returned error can't find the container with id 8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca Jul 07 04:28:31.385545 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:31.385510 2560 generic.go:356] "Generic (PLEG): container finished" podID="472fd6bd-b968-488c-a923-45be7b2261a5" containerID="b1a88fdd3548689e7ca6d6c6c2426296080f4254e3e2a02cfb44eb5ff2bd5a1a" exitCode=0 Jul 07 04:28:31.385918 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:31.385569 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" event={"ID":"472fd6bd-b968-488c-a923-45be7b2261a5","Type":"ContainerDied","Data":"b1a88fdd3548689e7ca6d6c6c2426296080f4254e3e2a02cfb44eb5ff2bd5a1a"} Jul 07 04:28:31.385918 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:31.385618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" event={"ID":"472fd6bd-b968-488c-a923-45be7b2261a5","Type":"ContainerStarted","Data":"8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca"} Jul 07 04:28:32.391805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:32.391771 2560 generic.go:356] "Generic (PLEG): container finished" podID="472fd6bd-b968-488c-a923-45be7b2261a5" containerID="e69685b19ad0822b5df37267786741e4602d56168afa433ddb056ddb451420b9" exitCode=0 Jul 07 04:28:32.392280 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:32.391830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" event={"ID":"472fd6bd-b968-488c-a923-45be7b2261a5","Type":"ContainerDied","Data":"e69685b19ad0822b5df37267786741e4602d56168afa433ddb056ddb451420b9"} Jul 07 04:28:33.398896 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:33.398863 2560 generic.go:356] "Generic (PLEG): container finished" podID="472fd6bd-b968-488c-a923-45be7b2261a5" containerID="c8d99593d46ec8a3386d45c8e478047fa2604b932ef84152939a8c79c7cbb5d8" exitCode=0 Jul 07 04:28:33.399283 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:33.398902 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" event={"ID":"472fd6bd-b968-488c-a923-45be7b2261a5","Type":"ContainerDied","Data":"c8d99593d46ec8a3386d45c8e478047fa2604b932ef84152939a8c79c7cbb5d8"} Jul 07 04:28:34.531490 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.531464 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:34.600476 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.600441 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzrxd\" (UniqueName: \"kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd\") pod \"472fd6bd-b968-488c-a923-45be7b2261a5\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " Jul 07 04:28:34.600679 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.600502 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle\") pod \"472fd6bd-b968-488c-a923-45be7b2261a5\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " Jul 07 04:28:34.600679 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.600541 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util\") pod \"472fd6bd-b968-488c-a923-45be7b2261a5\" (UID: \"472fd6bd-b968-488c-a923-45be7b2261a5\") " Jul 07 04:28:34.601415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.601387 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle" (OuterVolumeSpecName: "bundle") pod "472fd6bd-b968-488c-a923-45be7b2261a5" (UID: "472fd6bd-b968-488c-a923-45be7b2261a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:28:34.602515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.602492 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd" (OuterVolumeSpecName: "kube-api-access-nzrxd") pod "472fd6bd-b968-488c-a923-45be7b2261a5" (UID: "472fd6bd-b968-488c-a923-45be7b2261a5"). InnerVolumeSpecName "kube-api-access-nzrxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:28:34.606512 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.606462 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util" (OuterVolumeSpecName: "util") pod "472fd6bd-b968-488c-a923-45be7b2261a5" (UID: "472fd6bd-b968-488c-a923-45be7b2261a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:28:34.701137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.701034 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:34.701137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.701074 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/472fd6bd-b968-488c-a923-45be7b2261a5-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:34.701137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:34.701083 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrxd\" (UniqueName: \"kubernetes.io/projected/472fd6bd-b968-488c-a923-45be7b2261a5-kube-api-access-nzrxd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:28:35.413129 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:35.413097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" event={"ID":"472fd6bd-b968-488c-a923-45be7b2261a5","Type":"ContainerDied","Data":"8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca"} Jul 07 04:28:35.413279 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:35.413164 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8758b3402474b09e59a2da1188a8ed0eba84f5c55e46227d9b023f074c2057ca" Jul 07 04:28:35.413279 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:35.413112 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwnh4d" Jul 07 04:28:46.157284 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:46.157245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:46.252883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:46.252848 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:46.772534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:46.772499 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.056671 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.056590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.163869 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.163832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.202498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202467 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k"] Jul 07 04:28:47.202763 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202751 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="util" Jul 07 04:28:47.202805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202764 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="util" Jul 07 04:28:47.202805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202775 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="pull" Jul 07 04:28:47.202805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202782 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="pull" Jul 07 04:28:47.202805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202794 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="extract" Jul 07 04:28:47.202805 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202799 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="extract" Jul 07 04:28:47.202951 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.202850 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="472fd6bd-b968-488c-a923-45be7b2261a5" containerName="extract" Jul 07 04:28:47.209226 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.209201 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.213147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.213079 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 04:28:47.213147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.213133 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 04:28:47.213326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.213226 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 04:28:47.213903 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.213882 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 04:28:47.213971 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.213962 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ftdh2\"" Jul 07 04:28:47.221084 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.221059 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k"] Jul 07 04:28:47.311990 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.311877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.311990 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.311941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.312260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.312047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.312260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.312086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcxm\" (UniqueName: \"kubernetes.io/projected/22a8a364-c538-47c5-80de-86bc163e6ea1-kube-api-access-hbcxm\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.312260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.312112 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.351635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.351603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.413391 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413353 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.413565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413405 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.413565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413427 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.413565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcxm\" (UniqueName: \"kubernetes.io/projected/22a8a364-c538-47c5-80de-86bc163e6ea1-kube-api-access-hbcxm\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.413565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.413783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.413763 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-manifests\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.415817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.415795 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22a8a364-c538-47c5-80de-86bc163e6ea1-tmp\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.415954 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.415938 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-webhook-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.416040 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.416022 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22a8a364-c538-47c5-80de-86bc163e6ea1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.421154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.421134 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcxm\" (UniqueName: \"kubernetes.io/projected/22a8a364-c538-47c5-80de-86bc163e6ea1-kube-api-access-hbcxm\") pod \"opendatahub-operator-controller-manager-787b7c66b5-fd69k\" (UID: \"22a8a364-c538-47c5-80de-86bc163e6ea1\") " pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.450886 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.450856 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.519690 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.519647 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:47.554666 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.554632 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.659354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.659323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:47.667048 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.667019 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k"] Jul 07 04:28:47.670302 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:28:47.670277 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a8a364_c538_47c5_80de_86bc163e6ea1.slice/crio-bd691d74186a5a244cbf64a9d08dc17217f6474eb356e765a1fc78d0b5eca895 WatchSource:0}: Error finding container bd691d74186a5a244cbf64a9d08dc17217f6474eb356e765a1fc78d0b5eca895: Status 404 returned error can't find the container with id bd691d74186a5a244cbf64a9d08dc17217f6474eb356e765a1fc78d0b5eca895 Jul 07 04:28:47.757929 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:47.757895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:48.468557 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:48.468519 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" event={"ID":"22a8a364-c538-47c5-80de-86bc163e6ea1","Type":"ContainerStarted","Data":"bd691d74186a5a244cbf64a9d08dc17217f6474eb356e765a1fc78d0b5eca895"} Jul 07 04:28:51.485069 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:51.485035 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" event={"ID":"22a8a364-c538-47c5-80de-86bc163e6ea1","Type":"ContainerStarted","Data":"c623061f52205d2e60febef64d046ac16073bdab407f173e0311c7c8b99b8aa3"} Jul 07 04:28:52.490616 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:52.490585 2560 generic.go:356] "Generic (PLEG): container finished" podID="22a8a364-c538-47c5-80de-86bc163e6ea1" containerID="c623061f52205d2e60febef64d046ac16073bdab407f173e0311c7c8b99b8aa3" exitCode=0 Jul 07 04:28:52.491055 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:52.490623 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" event={"ID":"22a8a364-c538-47c5-80de-86bc163e6ea1","Type":"ContainerDied","Data":"c623061f52205d2e60febef64d046ac16073bdab407f173e0311c7c8b99b8aa3"} Jul 07 04:28:53.497213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:53.497181 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" event={"ID":"22a8a364-c538-47c5-80de-86bc163e6ea1","Type":"ContainerStarted","Data":"af8adff80d7e36e51e3a2787ae6f4bdfda7ddb1bac7b8ec273c356ab7cadb784"} Jul 07 04:28:53.497600 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:53.497290 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:28:53.519612 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:53.519556 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" podStartSLOduration=2.236654393 podStartE2EDuration="6.519537891s" totalImagesPullingTime="3.271611898s" totalInitContainerRuntime="1.0112716s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:28:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:28:53.517602852 +0000 UTC m=+291.185021046" watchObservedRunningTime="2026-07-07 04:28:53.519537891 +0000 UTC m=+291.186956083" Jul 07 04:28:56.322522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:56.322480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:28:57.543730 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:28:57.543695 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:03.100723 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:03.100693 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:29:03.104623 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:03.104597 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:29:03.108566 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:03.108547 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 04:29:04.504302 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:04.504271 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-787b7c66b5-fd69k" Jul 07 04:29:06.128502 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.128449 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:06.241650 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.241612 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:06.720598 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.720566 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:06.771612 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.771579 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv"] Jul 07 04:29:06.774834 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.774816 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.777442 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.777421 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 04:29:06.777577 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.777422 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 04:29:06.777577 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.777455 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-txwqx\"" Jul 07 04:29:06.778615 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.778600 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 04:29:06.778714 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.778627 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 04:29:06.793512 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.793490 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv"] Jul 07 04:29:06.885761 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.885722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.885761 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.885758 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tmp\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.885954 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.885838 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bkz\" (UniqueName: \"kubernetes.io/projected/505c6df7-6d97-41cb-a430-2d93f65e9f6f-kube-api-access-p9bkz\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.897270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.897233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:06.898188 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.898171 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crmjc"] Jul 07 04:29:06.901415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.901401 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:06.904421 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.904403 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 04:29:06.904534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.904519 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jmgk4\"" Jul 07 04:29:06.914618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.914596 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crmjc"] Jul 07 04:29:06.986475 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.986399 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.986475 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.986435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tmp\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.986656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.986479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bkz\" (UniqueName: \"kubernetes.io/projected/505c6df7-6d97-41cb-a430-2d93f65e9f6f-kube-api-access-p9bkz\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.988604 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.988580 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tmp\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:06.988870 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:06.988853 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/505c6df7-6d97-41cb-a430-2d93f65e9f6f-tls-certs\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:07.002764 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.002741 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bkz\" (UniqueName: \"kubernetes.io/projected/505c6df7-6d97-41cb-a430-2d93f65e9f6f-kube-api-access-p9bkz\") pod \"kube-auth-proxy-7db6fd6878-lgpsv\" (UID: \"505c6df7-6d97-41cb-a430-2d93f65e9f6f\") " pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:07.024762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.024725 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:07.084518 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.084482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" Jul 07 04:29:07.087816 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.087794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cb3fd7-08fa-4b29-a5fd-2c5467036635-cert\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.087908 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.087840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gnf\" (UniqueName: \"kubernetes.io/projected/86cb3fd7-08fa-4b29-a5fd-2c5467036635-kube-api-access-v2gnf\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.132077 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.132047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:07.188655 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.188623 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cb3fd7-08fa-4b29-a5fd-2c5467036635-cert\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.188789 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.188671 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gnf\" (UniqueName: \"kubernetes.io/projected/86cb3fd7-08fa-4b29-a5fd-2c5467036635-kube-api-access-v2gnf\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.191533 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.191498 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86cb3fd7-08fa-4b29-a5fd-2c5467036635-cert\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.202391 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.202362 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gnf\" (UniqueName: \"kubernetes.io/projected/86cb3fd7-08fa-4b29-a5fd-2c5467036635-kube-api-access-v2gnf\") pod \"odh-model-controller-7998c67dc4-crmjc\" (UID: \"86cb3fd7-08fa-4b29-a5fd-2c5467036635\") " pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.211079 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.211051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:07.223756 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.223735 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv"] Jul 07 04:29:07.225983 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:29:07.225957 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505c6df7_6d97_41cb_a430_2d93f65e9f6f.slice/crio-be6666857dbc54acbf130ce6a415409e210f0240cdde9510be787ff41f6035f3 WatchSource:0}: Error finding container be6666857dbc54acbf130ce6a415409e210f0240cdde9510be787ff41f6035f3: Status 404 returned error can't find the container with id be6666857dbc54acbf130ce6a415409e210f0240cdde9510be787ff41f6035f3 Jul 07 04:29:07.228595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.228574 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:29:07.229592 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.229559 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:07.373876 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.373851 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-crmjc"] Jul 07 04:29:07.375181 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:29:07.375158 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cb3fd7_08fa_4b29_a5fd_2c5467036635.slice/crio-f79b62f96a153c4a70b8c11cbd1607c9d4c122a8ed8805705b05d4c41576c12d WatchSource:0}: Error finding container f79b62f96a153c4a70b8c11cbd1607c9d4c122a8ed8805705b05d4c41576c12d: Status 404 returned error can't find the container with id f79b62f96a153c4a70b8c11cbd1607c9d4c122a8ed8805705b05d4c41576c12d Jul 07 04:29:07.557549 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.557467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" event={"ID":"86cb3fd7-08fa-4b29-a5fd-2c5467036635","Type":"ContainerStarted","Data":"f79b62f96a153c4a70b8c11cbd1607c9d4c122a8ed8805705b05d4c41576c12d"} Jul 07 04:29:07.558562 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.558537 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" event={"ID":"505c6df7-6d97-41cb-a430-2d93f65e9f6f","Type":"ContainerStarted","Data":"be6666857dbc54acbf130ce6a415409e210f0240cdde9510be787ff41f6035f3"} Jul 07 04:29:07.734129 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.734097 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-25zzt"] Jul 07 04:29:07.738661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.738640 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:07.741580 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.741549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 04:29:07.755679 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.755651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-25zzt"] Jul 07 04:29:07.756560 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.756517 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:07.895435 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.895397 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef55b86-d4c8-4639-b219-fc153e8f18df-tls-certs\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:07.895627 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.895462 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf55\" (UniqueName: \"kubernetes.io/projected/bef55b86-d4c8-4639-b219-fc153e8f18df-kube-api-access-4pf55\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:07.896656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.896622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:07.996618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.996582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef55b86-d4c8-4639-b219-fc153e8f18df-tls-certs\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:07.996794 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.996626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf55\" (UniqueName: \"kubernetes.io/projected/bef55b86-d4c8-4639-b219-fc153e8f18df-kube-api-access-4pf55\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:07.999263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:07.999237 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef55b86-d4c8-4639-b219-fc153e8f18df-tls-certs\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:08.005814 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.005790 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf55\" (UniqueName: \"kubernetes.io/projected/bef55b86-d4c8-4639-b219-fc153e8f18df-kube-api-access-4pf55\") pod \"model-serving-api-54d89bd5fc-25zzt\" (UID: \"bef55b86-d4c8-4639-b219-fc153e8f18df\") " pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:08.048950 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.048914 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:08.208797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.208762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-25zzt"] Jul 07 04:29:08.212376 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:29:08.212339 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef55b86_d4c8_4639_b219_fc153e8f18df.slice/crio-7962947aca445e7f95cbe0570544d524cd9217a79fc1805d5b5b49baceadd1fc WatchSource:0}: Error finding container 7962947aca445e7f95cbe0570544d524cd9217a79fc1805d5b5b49baceadd1fc: Status 404 returned error can't find the container with id 7962947aca445e7f95cbe0570544d524cd9217a79fc1805d5b5b49baceadd1fc Jul 07 04:29:08.231529 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.231495 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:08.233181 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.233149 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:08.235952 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.235930 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.239582 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.239546 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 04:29:08.239717 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.239585 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 04:29:08.239886 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.239867 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 04:29:08.239959 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.239920 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8bvdr\"" Jul 07 04:29:08.239959 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.239874 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 04:29:08.251349 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.251045 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:08.299466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299433 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjgq\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299598 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299627 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.299919 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.299661 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.304881 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.304796 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:08.400416 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400364 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjgq\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400606 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400548 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400665 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400611 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400665 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400685 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400772 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.400863 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.400772 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.402551 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.402518 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.405403 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.405374 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.405570 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.405546 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.406223 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.406181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.408242 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.408219 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.412283 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.412217 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.413862 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.413830 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjgq\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq\") pod \"istiod-openshift-gateway-7574459656-2rmrl\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.551588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.551500 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:08.563469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.563441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" event={"ID":"bef55b86-d4c8-4639-b219-fc153e8f18df","Type":"ContainerStarted","Data":"7962947aca445e7f95cbe0570544d524cd9217a79fc1805d5b5b49baceadd1fc"} Jul 07 04:29:08.721230 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:08.721144 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:08.724957 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:29:08.724920 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26325c7c_a111_411c_8a3d_0be3846acade.slice/crio-e4a6403e50c3818833bf29e28599c300de1e815687c99b9b46489b0d1b6bf9e9 WatchSource:0}: Error finding container e4a6403e50c3818833bf29e28599c300de1e815687c99b9b46489b0d1b6bf9e9: Status 404 returned error can't find the container with id e4a6403e50c3818833bf29e28599c300de1e815687c99b9b46489b0d1b6bf9e9 Jul 07 04:29:09.231885 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:09.231788 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:09.570180 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:09.570139 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" event={"ID":"26325c7c-a111-411c-8a3d-0be3846acade","Type":"ContainerStarted","Data":"e4a6403e50c3818833bf29e28599c300de1e815687c99b9b46489b0d1b6bf9e9"} Jul 07 04:29:12.588023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.587962 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" event={"ID":"505c6df7-6d97-41cb-a430-2d93f65e9f6f","Type":"ContainerStarted","Data":"58c4e9f07e4aa3528f9e2276e236f90cc05cfc89ab32d1751adb5e00dd352452"} Jul 07 04:29:12.589844 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.589814 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" event={"ID":"bef55b86-d4c8-4639-b219-fc153e8f18df","Type":"ContainerStarted","Data":"f0295db68e71fff65fe6222e6dcc3aa3e2b0d6b06e47b19da014bfb931a2b82e"} Jul 07 04:29:12.589984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.589871 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:12.591519 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.591452 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" event={"ID":"86cb3fd7-08fa-4b29-a5fd-2c5467036635","Type":"ContainerStarted","Data":"9ea9bd51f5b0b2c89e2c62ba4634615470dd2dfa49727e37aa1297cfe1d62acd"} Jul 07 04:29:12.591519 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.591506 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:12.606491 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.606440 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7db6fd6878-lgpsv" podStartSLOduration=2.214211175 podStartE2EDuration="6.606419515s" totalImagesPullingTime="4.39220834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:12.605051112 +0000 UTC m=+310.272469306" watchObservedRunningTime="2026-07-07 04:29:12.606419515 +0000 UTC m=+310.273837706" Jul 07 04:29:12.625249 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.625187 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" podStartSLOduration=2.384469991 podStartE2EDuration="6.625164884s" totalImagesPullingTime="4.240694893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:12.623639911 +0000 UTC m=+310.291058105" watchObservedRunningTime="2026-07-07 04:29:12.625164884 +0000 UTC m=+310.292583078" Jul 07 04:29:12.644184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.644124 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" podStartSLOduration=2.238386217 podStartE2EDuration="5.644101813s" totalImagesPullingTime="3.405715596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:12.643050555 +0000 UTC m=+310.310468748" watchObservedRunningTime="2026-07-07 04:29:12.644101813 +0000 UTC m=+310.311520005" Jul 07 04:29:12.920593 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.920553 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 04:29:12.920693 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:12.920618 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 04:29:13.602473 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.602423 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" event={"ID":"26325c7c-a111-411c-8a3d-0be3846acade","Type":"ContainerStarted","Data":"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5"} Jul 07 04:29:13.603161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.603126 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:13.604274 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.604245 2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-2rmrl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 04:29:13.604397 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.604303 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" podUID="26325c7c-a111-411c-8a3d-0be3846acade" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 04:29:13.673222 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.673184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:13.765931 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:13.765895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:14.425435 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.425398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:14.608326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.608294 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:14.642904 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.642841 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" podStartSLOduration=2.450592958 podStartE2EDuration="6.642823232s" totalImagesPullingTime="4.192230274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:13.662834167 +0000 UTC m=+311.330252359" watchObservedRunningTime="2026-07-07 04:29:14.642823232 +0000 UTC m=+312.310241423" Jul 07 04:29:14.710211 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.710133 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:14.910981 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.910951 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql"] Jul 07 04:29:14.914459 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.914443 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:14.917871 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.917842 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 04:29:14.917871 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.917842 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 04:29:14.918857 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.918839 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-45rl8\"" Jul 07 04:29:14.919270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.919255 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 04:29:14.945135 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.945103 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql"] Jul 07 04:29:14.968755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.968665 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-metrics-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:14.968755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.968715 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23f6f193-32eb-499b-8697-55120fb3f90d-manager-config\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:14.968993 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.968816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6kl\" (UniqueName: \"kubernetes.io/projected/23f6f193-32eb-499b-8697-55120fb3f90d-kube-api-access-mx6kl\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:14.968993 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:14.968878 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.069917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.069881 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-metrics-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.069917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.069922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23f6f193-32eb-499b-8697-55120fb3f90d-manager-config\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.070164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.069956 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6kl\" (UniqueName: \"kubernetes.io/projected/23f6f193-32eb-499b-8697-55120fb3f90d-kube-api-access-mx6kl\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.070164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.069978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.070865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.070837 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/23f6f193-32eb-499b-8697-55120fb3f90d-manager-config\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.072540 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.072514 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.072649 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.072549 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/23f6f193-32eb-499b-8697-55120fb3f90d-metrics-cert\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.078903 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.078877 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6kl\" (UniqueName: \"kubernetes.io/projected/23f6f193-32eb-499b-8697-55120fb3f90d-kube-api-access-mx6kl\") pod \"lws-controller-manager-687876b94d-zp2ql\" (UID: \"23f6f193-32eb-499b-8697-55120fb3f90d\") " pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.177696 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.177659 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:15.224871 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.224789 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:15.379766 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.379730 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql"] Jul 07 04:29:15.381018 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:29:15.380978 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f6f193_32eb_499b_8697_55120fb3f90d.slice/crio-23c515dced2b6bcd2bb02681621f8a8bb583d2db14bf460d8f7b96b01905a718 WatchSource:0}: Error finding container 23c515dced2b6bcd2bb02681621f8a8bb583d2db14bf460d8f7b96b01905a718: Status 404 returned error can't find the container with id 23c515dced2b6bcd2bb02681621f8a8bb583d2db14bf460d8f7b96b01905a718 Jul 07 04:29:15.614736 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:15.614696 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" event={"ID":"23f6f193-32eb-499b-8697-55120fb3f90d","Type":"ContainerStarted","Data":"23c515dced2b6bcd2bb02681621f8a8bb583d2db14bf460d8f7b96b01905a718"} Jul 07 04:29:16.949075 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:16.949044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.047815 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.047777 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.164430 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.164397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.315746 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.315716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.458967 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.458929 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.648945 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.648861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:17.937765 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:17.937676 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:29:19.112495 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.112459 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:19.112902 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.112650 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" podUID="26325c7c-a111-411c-8a3d-0be3846acade" containerName="discovery" containerID="cri-o://8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5" gracePeriod=30 Jul 07 04:29:19.355137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.355109 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:19.408110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408030 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408070 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpjgq\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408106 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408153 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408183 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408299 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408337 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig\") pod \"26325c7c-a111-411c-8a3d-0be3846acade\" (UID: \"26325c7c-a111-411c-8a3d-0be3846acade\") " Jul 07 04:29:19.408588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.408564 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:29:19.410507 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410480 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs" (OuterVolumeSpecName: "local-certs") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:29:19.410664 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410636 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:29:19.410810 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410677 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts" (OuterVolumeSpecName: "cacerts") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:29:19.410810 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410724 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq" (OuterVolumeSpecName: "kube-api-access-bpjgq") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "kube-api-access-bpjgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:29:19.410810 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410762 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:29:19.410810 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.410778 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token" (OuterVolumeSpecName: "istio-token") pod "26325c7c-a111-411c-8a3d-0be3846acade" (UID: "26325c7c-a111-411c-8a3d-0be3846acade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:29:19.509656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509617 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-ca-configmap\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509651 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509656 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509666 2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-kubeconfig\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509676 2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26325c7c-a111-411c-8a3d-0be3846acade-local-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509687 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpjgq\" (UniqueName: \"kubernetes.io/projected/26325c7c-a111-411c-8a3d-0be3846acade-kube-api-access-bpjgq\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509699 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-istio-csr-dns-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.509898 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.509710 2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26325c7c-a111-411c-8a3d-0be3846acade-cacerts\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:29:19.635099 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.635063 2560 generic.go:356] "Generic (PLEG): container finished" podID="26325c7c-a111-411c-8a3d-0be3846acade" containerID="8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5" exitCode=0 Jul 07 04:29:19.635270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.635129 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" Jul 07 04:29:19.635270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.635154 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" event={"ID":"26325c7c-a111-411c-8a3d-0be3846acade","Type":"ContainerDied","Data":"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5"} Jul 07 04:29:19.635270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.635229 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl" event={"ID":"26325c7c-a111-411c-8a3d-0be3846acade","Type":"ContainerDied","Data":"e4a6403e50c3818833bf29e28599c300de1e815687c99b9b46489b0d1b6bf9e9"} Jul 07 04:29:19.635270 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.635249 2560 scope.go:117] "RemoveContainer" containerID="8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5" Jul 07 04:29:19.644321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.644304 2560 scope.go:117] "RemoveContainer" containerID="8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5" Jul 07 04:29:19.644594 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:29:19.644575 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5\": container with ID starting with 8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5 not found: ID does not exist" containerID="8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5" Jul 07 04:29:19.644647 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.644602 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5"} err="failed to get container status \"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5\": rpc error: code = NotFound desc = could not find container \"8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5\": container with ID starting with 8a45ee9c97ee6c06972df563127ccfde3bf3b2f77d3b64a2219e9a384147fbf5 not found: ID does not exist" Jul 07 04:29:19.656663 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.656641 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:19.661119 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:19.661068 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2rmrl"] Jul 07 04:29:21.236339 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:21.236310 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26325c7c-a111-411c-8a3d-0be3846acade" path="/var/lib/kubelet/pods/26325c7c-a111-411c-8a3d-0be3846acade/volumes" Jul 07 04:29:23.605373 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:23.605344 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-crmjc" Jul 07 04:29:23.607233 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:23.607216 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-25zzt" Jul 07 04:29:24.662220 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:24.662177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" event={"ID":"23f6f193-32eb-499b-8697-55120fb3f90d","Type":"ContainerStarted","Data":"ccc9b38184882da31a449f34b2f74f6af401f078c652338d9fb0b4354a838adc"} Jul 07 04:29:24.662711 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:24.662265 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:29:24.680359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:24.680299 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" podStartSLOduration=1.537033352 podStartE2EDuration="10.680281426s" totalImagesPullingTime="9.143248074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:29:24.679116533 +0000 UTC m=+322.346534726" watchObservedRunningTime="2026-07-07 04:29:24.680281426 +0000 UTC m=+322.347699618" Jul 07 04:29:35.669543 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:29:35.669510 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-687876b94d-zp2ql" Jul 07 04:30:07.570863 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:07.570829 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:07.677833 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:07.677802 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:07.777438 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:07.777398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:22.864353 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.864317 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8"] Jul 07 04:30:22.864731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.864672 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26325c7c-a111-411c-8a3d-0be3846acade" containerName="discovery" Jul 07 04:30:22.864731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.864682 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="26325c7c-a111-411c-8a3d-0be3846acade" containerName="discovery" Jul 07 04:30:22.864804 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.864739 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="26325c7c-a111-411c-8a3d-0be3846acade" containerName="discovery" Jul 07 04:30:22.870208 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.870186 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:22.873073 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.873052 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 04:30:22.873200 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.873149 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 04:30:22.874229 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.874200 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mlfbb\"" Jul 07 04:30:22.876936 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.876916 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8"] Jul 07 04:30:22.960321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.960279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:22.960499 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.960345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkh6\" (UniqueName: \"kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:22.960499 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:22.960396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.061069 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.061023 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkh6\" (UniqueName: \"kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.061205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.061115 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.061205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.061148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.061489 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.061471 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.061565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.061506 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.070195 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.070168 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkh6\" (UniqueName: \"kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.180192 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.180096 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:23.316116 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.316091 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8"] Jul 07 04:30:23.318262 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:30:23.318235 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff68d5_d38e_478f_8836_3c85754308a8.slice/crio-f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452 WatchSource:0}: Error finding container f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452: Status 404 returned error can't find the container with id f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452 Jul 07 04:30:23.667087 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.667049 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx"] Jul 07 04:30:23.670723 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.670707 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.679729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.679700 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx"] Jul 07 04:30:23.768376 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.768275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.768376 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.768325 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2flh\" (UniqueName: \"kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.768597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.768444 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.869365 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.869326 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2flh\" (UniqueName: \"kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.869846 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.869396 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.869846 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.869462 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.869846 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.869794 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.869975 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.869865 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.878261 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.878238 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2flh\" (UniqueName: \"kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:23.929184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.929151 2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerID="45213545b5fb8c4c6fe41b82e4e709f9cccb803746a67db2d92ed7c5d2ebf125" exitCode=0 Jul 07 04:30:23.929350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.929187 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerDied","Data":"45213545b5fb8c4c6fe41b82e4e709f9cccb803746a67db2d92ed7c5d2ebf125"} Jul 07 04:30:23.929350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.929242 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerStarted","Data":"f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452"} Jul 07 04:30:23.980179 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:23.980146 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:24.108237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.108210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx"] Jul 07 04:30:24.109709 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:30:24.109673 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cffe46c_9a71_4a8c_9c47_d02452a95b7c.slice/crio-af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a WatchSource:0}: Error finding container af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a: Status 404 returned error can't find the container with id af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a Jul 07 04:30:24.273293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.273251 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw"] Jul 07 04:30:24.276877 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.276859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.287063 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.287036 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw"] Jul 07 04:30:24.374366 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.374330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.374530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.374382 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkkx\" (UniqueName: \"kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.374530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.374452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.475819 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.475786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkkx\" (UniqueName: \"kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.476043 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.475852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.476043 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.475903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.476294 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.476272 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.476362 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.476296 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.484610 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.484587 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkkx\" (UniqueName: \"kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.586208 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.586166 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:24.666324 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.666292 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x"] Jul 07 04:30:24.670543 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.670518 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.678995 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.678963 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x"] Jul 07 04:30:24.722420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.722397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw"] Jul 07 04:30:24.741134 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:30:24.741099 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode954f984_8d91_41d3_89e4_0dde5003a4f8.slice/crio-b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33 WatchSource:0}: Error finding container b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33: Status 404 returned error can't find the container with id b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33 Jul 07 04:30:24.779488 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.779456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.779639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.779553 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.779639 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.779577 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcwm\" (UniqueName: \"kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.880586 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.880562 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.880997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.880669 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.880997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.880688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcwm\" (UniqueName: \"kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.880997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.880984 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.880997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.881054 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.891708 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.891680 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcwm\" (UniqueName: \"kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:24.935998 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.935964 2560 generic.go:356] "Generic (PLEG): container finished" podID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerID="4e9df4aba4666d247d94737f0e7b8533bb03230038867eefc272f29b7c5e75e3" exitCode=0 Jul 07 04:30:24.936162 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.936035 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" event={"ID":"e954f984-8d91-41d3-89e4-0dde5003a4f8","Type":"ContainerDied","Data":"4e9df4aba4666d247d94737f0e7b8533bb03230038867eefc272f29b7c5e75e3"} Jul 07 04:30:24.936162 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.936085 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" event={"ID":"e954f984-8d91-41d3-89e4-0dde5003a4f8","Type":"ContainerStarted","Data":"b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33"} Jul 07 04:30:24.938171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.938148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerStarted","Data":"64088edfd77c8445f558cb382c541fd653fe612143f24d53fd3e79962bca9084"} Jul 07 04:30:24.939782 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.939758 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerID="fe9b4f3dc3c47161d561676be29ea3359fbdd19194c4f4c2b08b45a1c7e2257b" exitCode=0 Jul 07 04:30:24.939911 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.939800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" event={"ID":"0cffe46c-9a71-4a8c-9c47-d02452a95b7c","Type":"ContainerDied","Data":"fe9b4f3dc3c47161d561676be29ea3359fbdd19194c4f4c2b08b45a1c7e2257b"} Jul 07 04:30:24.939911 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.939835 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" event={"ID":"0cffe46c-9a71-4a8c-9c47-d02452a95b7c","Type":"ContainerStarted","Data":"af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a"} Jul 07 04:30:24.983264 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:24.983228 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:25.111977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.111951 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x"] Jul 07 04:30:25.113993 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:30:25.113969 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fba0229_5715_4a47_8ff7_4e921363c525.slice/crio-a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f WatchSource:0}: Error finding container a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f: Status 404 returned error can't find the container with id a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f Jul 07 04:30:25.946065 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.946041 2560 generic.go:356] "Generic (PLEG): container finished" podID="6fba0229-5715-4a47-8ff7-4e921363c525" containerID="03bc672879a5f2bd91585c6ad6ce41865d8c710e10fc6c327b831cb3c1746049" exitCode=0 Jul 07 04:30:25.946410 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.946120 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" event={"ID":"6fba0229-5715-4a47-8ff7-4e921363c525","Type":"ContainerDied","Data":"03bc672879a5f2bd91585c6ad6ce41865d8c710e10fc6c327b831cb3c1746049"} Jul 07 04:30:25.946410 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.946171 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" event={"ID":"6fba0229-5715-4a47-8ff7-4e921363c525","Type":"ContainerStarted","Data":"a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f"} Jul 07 04:30:25.948058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.947977 2560 generic.go:356] "Generic (PLEG): container finished" podID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerID="fcd07ae6202cf1b309c9c0d12e8868410a32788d4d3b0618d86b1e0f9863e929" exitCode=0 Jul 07 04:30:25.948058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.948030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" event={"ID":"e954f984-8d91-41d3-89e4-0dde5003a4f8","Type":"ContainerDied","Data":"fcd07ae6202cf1b309c9c0d12e8868410a32788d4d3b0618d86b1e0f9863e929"} Jul 07 04:30:25.950116 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.949966 2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerID="64088edfd77c8445f558cb382c541fd653fe612143f24d53fd3e79962bca9084" exitCode=0 Jul 07 04:30:25.950116 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:25.950026 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerDied","Data":"64088edfd77c8445f558cb382c541fd653fe612143f24d53fd3e79962bca9084"} Jul 07 04:30:26.957937 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.957913 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerID="199dc47e1ef79379892a97d3aa9ddc71124f894b30fbcc9d164aae82ff62e646" exitCode=0 Jul 07 04:30:26.958320 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.957995 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" event={"ID":"0cffe46c-9a71-4a8c-9c47-d02452a95b7c","Type":"ContainerDied","Data":"199dc47e1ef79379892a97d3aa9ddc71124f894b30fbcc9d164aae82ff62e646"} Jul 07 04:30:26.961944 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.961925 2560 generic.go:356] "Generic (PLEG): container finished" podID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerID="9482fb0b1fdfac953300680ee32aed59c91921c683d62735c27a4b8141251ba1" exitCode=0 Jul 07 04:30:26.962053 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.961976 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" event={"ID":"e954f984-8d91-41d3-89e4-0dde5003a4f8","Type":"ContainerDied","Data":"9482fb0b1fdfac953300680ee32aed59c91921c683d62735c27a4b8141251ba1"} Jul 07 04:30:26.971235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.971211 2560 generic.go:356] "Generic (PLEG): container finished" podID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerID="c89137a226b632790747c3348e70fd6227f00f13d3fadcf18a4f2e7e9cdac506" exitCode=0 Jul 07 04:30:26.971351 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:26.971272 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerDied","Data":"c89137a226b632790747c3348e70fd6227f00f13d3fadcf18a4f2e7e9cdac506"} Jul 07 04:30:27.978637 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:27.978607 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerID="96aba12c1545be330478447e528af39d5dfc1bc777e7e76b4958d91052270154" exitCode=0 Jul 07 04:30:27.979072 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:27.978676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" event={"ID":"0cffe46c-9a71-4a8c-9c47-d02452a95b7c","Type":"ContainerDied","Data":"96aba12c1545be330478447e528af39d5dfc1bc777e7e76b4958d91052270154"} Jul 07 04:30:27.980377 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:27.980355 2560 generic.go:356] "Generic (PLEG): container finished" podID="6fba0229-5715-4a47-8ff7-4e921363c525" containerID="6d10ec5aba73679c081a34b2dda1fadbbc469cec6430b04d865f7cade90c716e" exitCode=0 Jul 07 04:30:27.980556 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:27.980412 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" event={"ID":"6fba0229-5715-4a47-8ff7-4e921363c525","Type":"ContainerDied","Data":"6d10ec5aba73679c081a34b2dda1fadbbc469cec6430b04d865f7cade90c716e"} Jul 07 04:30:28.117985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.117957 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:28.151485 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.151461 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:28.213147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.213117 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdkkx\" (UniqueName: \"kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx\") pod \"e954f984-8d91-41d3-89e4-0dde5003a4f8\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " Jul 07 04:30:28.213310 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.213173 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle\") pod \"e954f984-8d91-41d3-89e4-0dde5003a4f8\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " Jul 07 04:30:28.213310 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.213247 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util\") pod \"e954f984-8d91-41d3-89e4-0dde5003a4f8\" (UID: \"e954f984-8d91-41d3-89e4-0dde5003a4f8\") " Jul 07 04:30:28.213695 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.213667 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle" (OuterVolumeSpecName: "bundle") pod "e954f984-8d91-41d3-89e4-0dde5003a4f8" (UID: "e954f984-8d91-41d3-89e4-0dde5003a4f8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:28.215399 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.215368 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx" (OuterVolumeSpecName: "kube-api-access-hdkkx") pod "e954f984-8d91-41d3-89e4-0dde5003a4f8" (UID: "e954f984-8d91-41d3-89e4-0dde5003a4f8"). InnerVolumeSpecName "kube-api-access-hdkkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:30:28.218877 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.218857 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util" (OuterVolumeSpecName: "util") pod "e954f984-8d91-41d3-89e4-0dde5003a4f8" (UID: "e954f984-8d91-41d3-89e4-0dde5003a4f8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:28.314488 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314459 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle\") pod \"e2ff68d5-d38e-478f-8836-3c85754308a8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " Jul 07 04:30:28.314640 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314557 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util\") pod \"e2ff68d5-d38e-478f-8836-3c85754308a8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " Jul 07 04:30:28.314640 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314599 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bkh6\" (UniqueName: \"kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6\") pod \"e2ff68d5-d38e-478f-8836-3c85754308a8\" (UID: \"e2ff68d5-d38e-478f-8836-3c85754308a8\") " Jul 07 04:30:28.314845 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314829 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.314883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314854 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkkx\" (UniqueName: \"kubernetes.io/projected/e954f984-8d91-41d3-89e4-0dde5003a4f8-kube-api-access-hdkkx\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.314883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.314871 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e954f984-8d91-41d3-89e4-0dde5003a4f8-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.315079 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.315055 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle" (OuterVolumeSpecName: "bundle") pod "e2ff68d5-d38e-478f-8836-3c85754308a8" (UID: "e2ff68d5-d38e-478f-8836-3c85754308a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:28.316692 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.316665 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6" (OuterVolumeSpecName: "kube-api-access-8bkh6") pod "e2ff68d5-d38e-478f-8836-3c85754308a8" (UID: "e2ff68d5-d38e-478f-8836-3c85754308a8"). InnerVolumeSpecName "kube-api-access-8bkh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:30:28.319841 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.319818 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util" (OuterVolumeSpecName: "util") pod "e2ff68d5-d38e-478f-8836-3c85754308a8" (UID: "e2ff68d5-d38e-478f-8836-3c85754308a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:28.415914 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.415880 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.415914 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.415906 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2ff68d5-d38e-478f-8836-3c85754308a8-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.415914 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.415916 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bkh6\" (UniqueName: \"kubernetes.io/projected/e2ff68d5-d38e-478f-8836-3c85754308a8-kube-api-access-8bkh6\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:28.988048 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.987990 2560 generic.go:356] "Generic (PLEG): container finished" podID="6fba0229-5715-4a47-8ff7-4e921363c525" containerID="0f9aa724b26643c6e25680cfb7e758fc177ae0a0356c466c986fd526b91c4430" exitCode=0 Jul 07 04:30:28.988048 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.988038 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" event={"ID":"6fba0229-5715-4a47-8ff7-4e921363c525","Type":"ContainerDied","Data":"0f9aa724b26643c6e25680cfb7e758fc177ae0a0356c466c986fd526b91c4430"} Jul 07 04:30:28.989802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.989784 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" Jul 07 04:30:28.989802 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.989795 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw" event={"ID":"e954f984-8d91-41d3-89e4-0dde5003a4f8","Type":"ContainerDied","Data":"b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33"} Jul 07 04:30:28.989936 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.989832 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c8c97d7136a2425edf7eec68c21109a2ebbd9f0fece5136032d02a866c6e33" Jul 07 04:30:28.991542 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.991518 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" Jul 07 04:30:28.991660 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.991560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8" event={"ID":"e2ff68d5-d38e-478f-8836-3c85754308a8","Type":"ContainerDied","Data":"f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452"} Jul 07 04:30:28.991660 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:28.991595 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f38f58a97bf284856e12913e29b30523284420a081e8d002ee90937f939e9452" Jul 07 04:30:29.117515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.117490 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:29.221930 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.221892 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle\") pod \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " Jul 07 04:30:29.222122 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.221941 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2flh\" (UniqueName: \"kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh\") pod \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " Jul 07 04:30:29.222122 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.222046 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util\") pod \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\" (UID: \"0cffe46c-9a71-4a8c-9c47-d02452a95b7c\") " Jul 07 04:30:29.222414 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.222387 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle" (OuterVolumeSpecName: "bundle") pod "0cffe46c-9a71-4a8c-9c47-d02452a95b7c" (UID: "0cffe46c-9a71-4a8c-9c47-d02452a95b7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:29.224121 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.224094 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh" (OuterVolumeSpecName: "kube-api-access-m2flh") pod "0cffe46c-9a71-4a8c-9c47-d02452a95b7c" (UID: "0cffe46c-9a71-4a8c-9c47-d02452a95b7c"). InnerVolumeSpecName "kube-api-access-m2flh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:30:29.227322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.227301 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util" (OuterVolumeSpecName: "util") pod "0cffe46c-9a71-4a8c-9c47-d02452a95b7c" (UID: "0cffe46c-9a71-4a8c-9c47-d02452a95b7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:29.323597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.323559 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2flh\" (UniqueName: \"kubernetes.io/projected/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-kube-api-access-m2flh\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:29.323597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.323598 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:29.323597 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.323608 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0cffe46c-9a71-4a8c-9c47-d02452a95b7c-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:29.998630 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.998593 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" event={"ID":"0cffe46c-9a71-4a8c-9c47-d02452a95b7c","Type":"ContainerDied","Data":"af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a"} Jul 07 04:30:29.999080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.998633 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx" Jul 07 04:30:29.999080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:29.998671 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af63ae065a71f7d6e25e85c88418fc6e04a70eace626454a062a9233e0e0640a" Jul 07 04:30:30.127028 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.126992 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:30.230904 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.230868 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle\") pod \"6fba0229-5715-4a47-8ff7-4e921363c525\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " Jul 07 04:30:30.231107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.230963 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util\") pod \"6fba0229-5715-4a47-8ff7-4e921363c525\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " Jul 07 04:30:30.231107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.230994 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcwm\" (UniqueName: \"kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm\") pod \"6fba0229-5715-4a47-8ff7-4e921363c525\" (UID: \"6fba0229-5715-4a47-8ff7-4e921363c525\") " Jul 07 04:30:30.231616 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.231570 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle" (OuterVolumeSpecName: "bundle") pod "6fba0229-5715-4a47-8ff7-4e921363c525" (UID: "6fba0229-5715-4a47-8ff7-4e921363c525"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:30.233033 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.232986 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm" (OuterVolumeSpecName: "kube-api-access-xfcwm") pod "6fba0229-5715-4a47-8ff7-4e921363c525" (UID: "6fba0229-5715-4a47-8ff7-4e921363c525"). InnerVolumeSpecName "kube-api-access-xfcwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:30:30.236292 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.236259 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util" (OuterVolumeSpecName: "util") pod "6fba0229-5715-4a47-8ff7-4e921363c525" (UID: "6fba0229-5715-4a47-8ff7-4e921363c525"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:30:30.332583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.332548 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-util\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:30.332583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.332580 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcwm\" (UniqueName: \"kubernetes.io/projected/6fba0229-5715-4a47-8ff7-4e921363c525-kube-api-access-xfcwm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:30.332774 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:30.332590 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fba0229-5715-4a47-8ff7-4e921363c525-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:30:31.005420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:31.005381 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" event={"ID":"6fba0229-5715-4a47-8ff7-4e921363c525","Type":"ContainerDied","Data":"a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f"} Jul 07 04:30:31.005420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:31.005406 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x" Jul 07 04:30:31.005420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:31.005429 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6193e0791ed0771a3009d3e7181f7ca6862c6b2bfe82d99865fa55574cadb5f" Jul 07 04:30:31.976707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:31.976673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:32.047479 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:32.047443 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:35.380566 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:35.380535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:36.378911 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:36.378877 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:41.283545 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:41.283503 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:42.444346 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:42.444307 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:30:42.729042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:42.728937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:42.874583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:42.874551 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:44.262200 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:44.262165 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:44.353120 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:44.353083 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:45.440675 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:45.440636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:46.608319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:46.608282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:46.695172 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:46.695130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:47.652599 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:47.652561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:49.196194 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:49.196160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:49.296041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:49.295990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:52.781712 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:52.781678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:52.838845 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:52.838814 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:53.578571 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:53.578545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:53.633216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:53.633181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:55.433617 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433586 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr"] Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433918 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="extract" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433929 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="extract" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433938 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="util" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433944 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="util" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433950 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="util" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433956 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="util" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433962 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="extract" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433967 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="extract" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433974 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="pull" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433979 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="pull" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433985 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="pull" Jul 07 04:30:55.433985 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.433990 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="pull" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434017 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434023 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434031 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="pull" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434036 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="pull" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434041 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="pull" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434045 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="pull" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434051 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="util" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434056 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="util" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434062 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="util" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434067 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="util" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434074 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434079 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434128 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ff68d5-d38e-478f-8836-3c85754308a8" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434136 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fba0229-5715-4a47-8ff7-4e921363c525" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434143 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cffe46c-9a71-4a8c-9c47-d02452a95b7c" containerName="extract" Jul 07 04:30:55.434367 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.434149 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e954f984-8d91-41d3-89e4-0dde5003a4f8" containerName="extract" Jul 07 04:30:55.437060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.437046 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.439933 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.439902 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 04:30:55.440095 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.440054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 04:30:55.440182 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.440102 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 04:30:55.440182 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.440138 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 04:30:55.440864 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.440842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:55.441053 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.441037 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mlfbb\"" Jul 07 04:30:55.452051 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.452029 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr"] Jul 07 04:30:55.540523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.540483 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48s4z\" (UniqueName: \"kubernetes.io/projected/588265d9-beb7-4dfb-8105-b742f2ccafab-kube-api-access-48s4z\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.540674 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.540582 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.540674 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.540620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/588265d9-beb7-4dfb-8105-b742f2ccafab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.641304 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.641269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48s4z\" (UniqueName: \"kubernetes.io/projected/588265d9-beb7-4dfb-8105-b742f2ccafab-kube-api-access-48s4z\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.641479 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.641338 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.641479 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.641357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/588265d9-beb7-4dfb-8105-b742f2ccafab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.641479 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:30:55.641472 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 07 04:30:55.641614 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:30:55.641553 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert podName:588265d9-beb7-4dfb-8105-b742f2ccafab nodeName:}" failed. No retries permitted until 2026-07-07 04:30:56.141533996 +0000 UTC m=+413.808952171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-9ngqr" (UID: "588265d9-beb7-4dfb-8105-b742f2ccafab") : secret "plugin-serving-cert" not found Jul 07 04:30:55.641945 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.641929 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/588265d9-beb7-4dfb-8105-b742f2ccafab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:55.652322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:55.652291 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48s4z\" (UniqueName: \"kubernetes.io/projected/588265d9-beb7-4dfb-8105-b742f2ccafab-kube-api-access-48s4z\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:56.011795 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:56.011756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:56.146596 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:56.146559 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:56.149058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:56.149037 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/588265d9-beb7-4dfb-8105-b742f2ccafab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9ngqr\" (UID: \"588265d9-beb7-4dfb-8105-b742f2ccafab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:56.346454 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:56.346429 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" Jul 07 04:30:56.475405 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:56.475372 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr"] Jul 07 04:30:56.478564 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:30:56.478535 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod588265d9_beb7_4dfb_8105_b742f2ccafab.slice/crio-37a99ad6375a893a162b95106b158a482bd572588fe98329b9bed894e124b1b9 WatchSource:0}: Error finding container 37a99ad6375a893a162b95106b158a482bd572588fe98329b9bed894e124b1b9: Status 404 returned error can't find the container with id 37a99ad6375a893a162b95106b158a482bd572588fe98329b9bed894e124b1b9 Jul 07 04:30:57.027946 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:57.027906 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:57.088432 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:57.088397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:57.133871 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:57.133832 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" event={"ID":"588265d9-beb7-4dfb-8105-b742f2ccafab","Type":"ContainerStarted","Data":"37a99ad6375a893a162b95106b158a482bd572588fe98329b9bed894e124b1b9"} Jul 07 04:30:57.379061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:57.379021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:30:57.478770 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:30:57.478737 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:03.457326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:03.457295 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:03.557716 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:03.557685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:07.464655 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:07.464601 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-75fc44b88d-wcv56" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerName="console" containerID="cri-o://7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937" gracePeriod=15 Jul 07 04:31:08.215633 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:08.215572 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:08.324075 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:08.324037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:12.201046 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:12.200968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:12.300309 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:12.300281 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:12.787734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:12.787697 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:12.842725 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:12.842681 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:14.980541 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:14.980500 2560 patch_prober.go:28] interesting pod/console-75fc44b88d-wcv56 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body= Jul 07 04:31:14.980941 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:14.980561 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-75fc44b88d-wcv56" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" Jul 07 04:31:16.392222 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:16.392179 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:16.492336 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:16.492296 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:19.834536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.834513 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:31:19.975618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975586 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975620 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975675 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975732 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.975853 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.975759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqs9h\" (UniqueName: \"kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h\") pod \"e90bf5a8-b0de-4785-8186-84dad4ec9008\" (UID: \"e90bf5a8-b0de-4785-8186-84dad4ec9008\") " Jul 07 04:31:19.976160 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.976078 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca" (OuterVolumeSpecName: "service-ca") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:31:19.976300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.976272 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:31:19.976540 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.976340 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config" (OuterVolumeSpecName: "console-config") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:31:19.976540 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.976372 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 04:31:19.978015 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.977987 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h" (OuterVolumeSpecName: "kube-api-access-bqs9h") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "kube-api-access-bqs9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:31:19.978299 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.978280 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:31:19.978372 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:19.978298 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e90bf5a8-b0de-4785-8186-84dad4ec9008" (UID: "e90bf5a8-b0de-4785-8186-84dad4ec9008"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:31:20.076702 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076657 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqs9h\" (UniqueName: \"kubernetes.io/projected/e90bf5a8-b0de-4785-8186-84dad4ec9008-kube-api-access-bqs9h\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076702 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076693 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076702 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076702 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076702 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076711 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076976 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076720 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076976 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076727 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e90bf5a8-b0de-4785-8186-84dad4ec9008-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.076976 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.076736 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e90bf5a8-b0de-4785-8186-84dad4ec9008-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:31:20.268395 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.268309 2560 generic.go:356] "Generic (PLEG): container finished" podID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerID="7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937" exitCode=0 Jul 07 04:31:20.268395 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.268354 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fc44b88d-wcv56" Jul 07 04:31:20.268611 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.268393 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fc44b88d-wcv56" event={"ID":"e90bf5a8-b0de-4785-8186-84dad4ec9008","Type":"ContainerDied","Data":"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937"} Jul 07 04:31:20.268611 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.268445 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fc44b88d-wcv56" event={"ID":"e90bf5a8-b0de-4785-8186-84dad4ec9008","Type":"ContainerDied","Data":"f9db41f3620ece59927ba89bed45b9a8e55314a2eb7ff8b7d1f205908c5c9be8"} Jul 07 04:31:20.268611 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.268468 2560 scope.go:117] "RemoveContainer" containerID="7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937" Jul 07 04:31:20.270120 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.270092 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" event={"ID":"588265d9-beb7-4dfb-8105-b742f2ccafab","Type":"ContainerStarted","Data":"6dd06e8780127344eeaec069c6b27879693176bf9cfb48ff3da20a28bd25a0fc"} Jul 07 04:31:20.279755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.278590 2560 scope.go:117] "RemoveContainer" containerID="7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937" Jul 07 04:31:20.280310 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:31:20.280287 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937\": container with ID starting with 7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937 not found: ID does not exist" containerID="7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937" Jul 07 04:31:20.280386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.280316 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937"} err="failed to get container status \"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937\": rpc error: code = NotFound desc = could not find container \"7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937\": container with ID starting with 7a4170661d23771c239729beb26398345d51e6417200d656648311c9aa463937 not found: ID does not exist" Jul 07 04:31:20.293394 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.293354 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9ngqr" podStartSLOduration=1.961226641 podStartE2EDuration="25.293342346s" totalImagesPullingTime="23.332115705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:31:20.29146285 +0000 UTC m=+437.958881044" watchObservedRunningTime="2026-07-07 04:31:20.293342346 +0000 UTC m=+437.960760539" Jul 07 04:31:20.312490 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.312468 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:31:20.319545 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:20.319522 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75fc44b88d-wcv56"] Jul 07 04:31:21.236380 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:21.236349 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" path="/var/lib/kubelet/pods/e90bf5a8-b0de-4785-8186-84dad4ec9008/volumes" Jul 07 04:31:22.456518 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:22.456479 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:22.552926 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:22.552893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:27.211040 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:27.210940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:27.312698 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:27.312668 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:34.994758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:34.994723 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:35.350842 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:35.350807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:35.576269 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:35.576240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:51.891056 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:51.891021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:52.002977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.002945 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:52.058664 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.058633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:52.202459 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.202376 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:31:52.202716 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.202704 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerName="console" Jul 07 04:31:52.202760 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.202718 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerName="console" Jul 07 04:31:52.202800 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.202791 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90bf5a8-b0de-4785-8186-84dad4ec9008" containerName="console" Jul 07 04:31:52.273291 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.273231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.273495 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.273258 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:31:52.273495 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.273357 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:31:52.275568 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.275549 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 04:31:52.348017 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.347962 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a6f4e9d-88b6-4b74-8641-3563c7609168-config-file\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.348246 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.348053 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnn6\" (UniqueName: \"kubernetes.io/projected/1a6f4e9d-88b6-4b74-8641-3563c7609168-kube-api-access-rlnn6\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.448734 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.448699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a6f4e9d-88b6-4b74-8641-3563c7609168-config-file\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.448932 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.448782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnn6\" (UniqueName: \"kubernetes.io/projected/1a6f4e9d-88b6-4b74-8641-3563c7609168-kube-api-access-rlnn6\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.449430 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.449405 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1a6f4e9d-88b6-4b74-8641-3563c7609168-config-file\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.461269 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.461206 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnn6\" (UniqueName: \"kubernetes.io/projected/1a6f4e9d-88b6-4b74-8641-3563c7609168-kube-api-access-rlnn6\") pod \"limitador-limitador-54d6d988d5-7mv4m\" (UID: \"1a6f4e9d-88b6-4b74-8641-3563c7609168\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.583690 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.583651 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:52.706240 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.706209 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:31:52.710636 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.710620 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:31:52.713260 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.713211 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q6hdh\"" Jul 07 04:31:52.718019 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.717976 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:31:52.719959 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:31:52.719938 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6f4e9d_88b6_4b74_8641_3563c7609168.slice/crio-7f4efcb598e1676e55ee77e03792317b897c0f69202b8a011e41bd7021c38ecd WatchSource:0}: Error finding container 7f4efcb598e1676e55ee77e03792317b897c0f69202b8a011e41bd7021c38ecd: Status 404 returned error can't find the container with id 7f4efcb598e1676e55ee77e03792317b897c0f69202b8a011e41bd7021c38ecd Jul 07 04:31:52.720844 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.720821 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:31:52.749201 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.749167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:52.820766 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.820738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:31:52.824725 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.824701 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:31:52.828338 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.828323 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:31:52.833649 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.833626 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:31:52.851839 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.851816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbjj\" (UniqueName: \"kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj\") pod \"authorino-58cd955675-smm8w\" (UID: \"95281795-139a-4172-827f-0bc67e80d651\") " pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:31:52.953254 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.953227 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tch27\" (UniqueName: \"kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27\") pod \"authorino-75d556bd77-c4dgw\" (UID: \"90dc0569-925c-42a1-8f76-ff0b72d42e84\") " pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:31:52.953577 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.953307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbjj\" (UniqueName: \"kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj\") pod \"authorino-58cd955675-smm8w\" (UID: \"95281795-139a-4172-827f-0bc67e80d651\") " pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:31:52.960825 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:52.960803 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbjj\" (UniqueName: \"kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj\") pod \"authorino-58cd955675-smm8w\" (UID: \"95281795-139a-4172-827f-0bc67e80d651\") " pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:31:53.028678 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.028610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:31:53.053758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.053734 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tch27\" (UniqueName: \"kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27\") pod \"authorino-75d556bd77-c4dgw\" (UID: \"90dc0569-925c-42a1-8f76-ff0b72d42e84\") " pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:31:53.061661 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.061638 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tch27\" (UniqueName: \"kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27\") pod \"authorino-75d556bd77-c4dgw\" (UID: \"90dc0569-925c-42a1-8f76-ff0b72d42e84\") " pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:31:53.137588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.137552 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:31:53.155882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.155861 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:31:53.157980 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:31:53.157952 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95281795_139a_4172_827f_0bc67e80d651.slice/crio-a54b8bc9ce1e4a8d32b20de373aedfe0cc9a97c8af9d48ab0c59304bcaa3254e WatchSource:0}: Error finding container a54b8bc9ce1e4a8d32b20de373aedfe0cc9a97c8af9d48ab0c59304bcaa3254e: Status 404 returned error can't find the container with id a54b8bc9ce1e4a8d32b20de373aedfe0cc9a97c8af9d48ab0c59304bcaa3254e Jul 07 04:31:53.303975 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.303842 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:31:53.308254 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:31:53.308228 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90dc0569_925c_42a1_8f76_ff0b72d42e84.slice/crio-1d39687d79bbd535b799750e22816073acdba61cc8e3dc978c2e095c98f8add2 WatchSource:0}: Error finding container 1d39687d79bbd535b799750e22816073acdba61cc8e3dc978c2e095c98f8add2: Status 404 returned error can't find the container with id 1d39687d79bbd535b799750e22816073acdba61cc8e3dc978c2e095c98f8add2 Jul 07 04:31:53.430384 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.430340 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" event={"ID":"1a6f4e9d-88b6-4b74-8641-3563c7609168","Type":"ContainerStarted","Data":"7f4efcb598e1676e55ee77e03792317b897c0f69202b8a011e41bd7021c38ecd"} Jul 07 04:31:53.432957 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.432925 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-smm8w" event={"ID":"95281795-139a-4172-827f-0bc67e80d651","Type":"ContainerStarted","Data":"a54b8bc9ce1e4a8d32b20de373aedfe0cc9a97c8af9d48ab0c59304bcaa3254e"} Jul 07 04:31:53.434828 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:53.434800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c4dgw" event={"ID":"90dc0569-925c-42a1-8f76-ff0b72d42e84","Type":"ContainerStarted","Data":"1d39687d79bbd535b799750e22816073acdba61cc8e3dc978c2e095c98f8add2"} Jul 07 04:31:58.466938 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.466903 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" event={"ID":"1a6f4e9d-88b6-4b74-8641-3563c7609168","Type":"ContainerStarted","Data":"af3ee9df3ed14d38d82915eda6c254f64e213e56284c2f0662af24578d224df9"} Jul 07 04:31:58.467455 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.466994 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:31:58.468434 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.468411 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-smm8w" event={"ID":"95281795-139a-4172-827f-0bc67e80d651","Type":"ContainerStarted","Data":"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e"} Jul 07 04:31:58.469736 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.469714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c4dgw" event={"ID":"90dc0569-925c-42a1-8f76-ff0b72d42e84","Type":"ContainerStarted","Data":"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc"} Jul 07 04:31:58.485393 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.485349 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" podStartSLOduration=1.5944148660000002 podStartE2EDuration="6.48533551s" totalImagesPullingTime="4.890920644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:31:58.483664814 +0000 UTC m=+476.151083007" watchObservedRunningTime="2026-07-07 04:31:58.48533551 +0000 UTC m=+476.152753702" Jul 07 04:31:58.500037 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.499975 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-c4dgw" podStartSLOduration=2.24935322 podStartE2EDuration="6.499961357s" totalImagesPullingTime="4.250608137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:31:58.49836404 +0000 UTC m=+476.165782231" watchObservedRunningTime="2026-07-07 04:31:58.499961357 +0000 UTC m=+476.167379557" Jul 07 04:31:58.522154 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.517073 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-smm8w" podStartSLOduration=2.080114002 podStartE2EDuration="6.51705596s" totalImagesPullingTime="4.436941958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:31:58.513452929 +0000 UTC m=+476.180871131" watchObservedRunningTime="2026-07-07 04:31:58.51705596 +0000 UTC m=+476.184474153" Jul 07 04:31:58.540081 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:31:58.540048 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:32:00.478670 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:00.478627 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-smm8w" podUID="95281795-139a-4172-827f-0bc67e80d651" containerName="authorino" containerID="cri-o://072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e" gracePeriod=30 Jul 07 04:32:00.725091 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:00.725067 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:32:00.822392 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:00.822363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sbjj\" (UniqueName: \"kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj\") pod \"95281795-139a-4172-827f-0bc67e80d651\" (UID: \"95281795-139a-4172-827f-0bc67e80d651\") " Jul 07 04:32:00.824448 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:00.824427 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj" (OuterVolumeSpecName: "kube-api-access-2sbjj") pod "95281795-139a-4172-827f-0bc67e80d651" (UID: "95281795-139a-4172-827f-0bc67e80d651"). InnerVolumeSpecName "kube-api-access-2sbjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:32:00.922908 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:00.922880 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbjj\" (UniqueName: \"kubernetes.io/projected/95281795-139a-4172-827f-0bc67e80d651-kube-api-access-2sbjj\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:32:01.484488 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.484454 2560 generic.go:356] "Generic (PLEG): container finished" podID="95281795-139a-4172-827f-0bc67e80d651" containerID="072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e" exitCode=0 Jul 07 04:32:01.484865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.484497 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-smm8w" Jul 07 04:32:01.484865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.484547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-smm8w" event={"ID":"95281795-139a-4172-827f-0bc67e80d651","Type":"ContainerDied","Data":"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e"} Jul 07 04:32:01.484865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.484585 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-smm8w" event={"ID":"95281795-139a-4172-827f-0bc67e80d651","Type":"ContainerDied","Data":"a54b8bc9ce1e4a8d32b20de373aedfe0cc9a97c8af9d48ab0c59304bcaa3254e"} Jul 07 04:32:01.484865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.484602 2560 scope.go:117] "RemoveContainer" containerID="072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e" Jul 07 04:32:01.493083 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.493066 2560 scope.go:117] "RemoveContainer" containerID="072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e" Jul 07 04:32:01.493355 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:32:01.493337 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e\": container with ID starting with 072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e not found: ID does not exist" containerID="072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e" Jul 07 04:32:01.493394 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.493365 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e"} err="failed to get container status \"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e\": rpc error: code = NotFound desc = could not find container \"072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e\": container with ID starting with 072f5381829ba00f41ab2dab7b7120dc430d74b0e965358fae0575ce3b12783e not found: ID does not exist" Jul 07 04:32:01.502799 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.502775 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:32:01.507078 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:01.507061 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-smm8w"] Jul 07 04:32:03.235954 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:03.235929 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95281795-139a-4172-827f-0bc67e80d651" path="/var/lib/kubelet/pods/95281795-139a-4172-827f-0bc67e80d651/volumes" Jul 07 04:32:09.475637 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:09.475606 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7mv4m" Jul 07 04:32:13.677227 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:13.677188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:26.214089 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.214054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:26.313419 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.313366 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:26.429238 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.429197 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:26.429622 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.429603 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95281795-139a-4172-827f-0bc67e80d651" containerName="authorino" Jul 07 04:32:26.429712 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.429625 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="95281795-139a-4172-827f-0bc67e80d651" containerName="authorino" Jul 07 04:32:26.429712 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.429707 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="95281795-139a-4172-827f-0bc67e80d651" containerName="authorino" Jul 07 04:32:26.432526 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.432502 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:26.442286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.442262 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:26.528543 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.528449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk8m\" (UniqueName: \"kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m\") pod \"authorino-5d4b67c446-7jvj9\" (UID: \"4973ec21-6e2b-4845-8ed8-c1b26643fbc3\") " pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:26.629629 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.629580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk8m\" (UniqueName: \"kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m\") pod \"authorino-5d4b67c446-7jvj9\" (UID: \"4973ec21-6e2b-4845-8ed8-c1b26643fbc3\") " pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:26.638490 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.638462 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk8m\" (UniqueName: \"kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m\") pod \"authorino-5d4b67c446-7jvj9\" (UID: \"4973ec21-6e2b-4845-8ed8-c1b26643fbc3\") " pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:26.697891 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.697853 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:26.698144 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.698131 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:26.723413 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.723380 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cfcc4fc7b-qqgjk"] Jul 07 04:32:26.728351 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.728331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:26.732665 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.732637 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cfcc4fc7b-qqgjk"] Jul 07 04:32:26.805096 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.805056 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cfcc4fc7b-qqgjk"] Jul 07 04:32:26.805523 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:32:26.805387 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9zjk2], unattached volumes=[], failed to process volumes=[kube-api-access-9zjk2]: context canceled" pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" podUID="441000c3-d6e5-4321-b90e-b374b4973406" Jul 07 04:32:26.831200 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.831160 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:32:26.831383 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.831225 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjk2\" (UniqueName: \"kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2\") pod \"authorino-cfcc4fc7b-qqgjk\" (UID: \"441000c3-d6e5-4321-b90e-b374b4973406\") " pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:26.834633 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.834614 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:26.837078 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.837054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 04:32:26.840817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.840750 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:26.842947 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:32:26.842918 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4973ec21_6e2b_4845_8ed8_c1b26643fbc3.slice/crio-91c10e7e6e9f36f9b7aa358f580c9c73dee02e1439a746d4b5d4ec46e997f785 WatchSource:0}: Error finding container 91c10e7e6e9f36f9b7aa358f580c9c73dee02e1439a746d4b5d4ec46e997f785: Status 404 returned error can't find the container with id 91c10e7e6e9f36f9b7aa358f580c9c73dee02e1439a746d4b5d4ec46e997f785 Jul 07 04:32:26.848296 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.848269 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:32:26.931766 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.931725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:26.931979 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.931780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmfq\" (UniqueName: \"kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:26.931979 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.931839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjk2\" (UniqueName: \"kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2\") pod \"authorino-cfcc4fc7b-qqgjk\" (UID: \"441000c3-d6e5-4321-b90e-b374b4973406\") " pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:26.941409 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:26.941380 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjk2\" (UniqueName: \"kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2\") pod \"authorino-cfcc4fc7b-qqgjk\" (UID: \"441000c3-d6e5-4321-b90e-b374b4973406\") " pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:27.032641 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.032603 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:27.032854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.032664 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmfq\" (UniqueName: \"kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:27.035111 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.035087 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:27.040915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.040849 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmfq\" (UniqueName: \"kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq\") pod \"authorino-7cfd88c964-8hqjf\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:27.153322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.153281 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:32:27.292345 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.292318 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:32:27.294867 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:32:27.294834 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0890db77_3afd_47d4_84e3_615a74adc81d.slice/crio-18c84d2da7b8349c9ece595c4e9b6bc5a6ba0c43a831654b9e482d3f606f8d86 WatchSource:0}: Error finding container 18c84d2da7b8349c9ece595c4e9b6bc5a6ba0c43a831654b9e482d3f606f8d86: Status 404 returned error can't find the container with id 18c84d2da7b8349c9ece595c4e9b6bc5a6ba0c43a831654b9e482d3f606f8d86 Jul 07 04:32:27.611890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.611854 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" event={"ID":"4973ec21-6e2b-4845-8ed8-c1b26643fbc3","Type":"ContainerStarted","Data":"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49"} Jul 07 04:32:27.612110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.611900 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" event={"ID":"4973ec21-6e2b-4845-8ed8-c1b26643fbc3","Type":"ContainerStarted","Data":"91c10e7e6e9f36f9b7aa358f580c9c73dee02e1439a746d4b5d4ec46e997f785"} Jul 07 04:32:27.612110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.611896 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" podUID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" containerName="authorino" containerID="cri-o://1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49" gracePeriod=30 Jul 07 04:32:27.613157 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.613136 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:27.613265 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.613132 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" event={"ID":"0890db77-3afd-47d4-84e3-615a74adc81d","Type":"ContainerStarted","Data":"18c84d2da7b8349c9ece595c4e9b6bc5a6ba0c43a831654b9e482d3f606f8d86"} Jul 07 04:32:27.620971 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.620952 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:27.631357 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.631306 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" podStartSLOduration=1.2621520990000001 podStartE2EDuration="1.631291181s" totalImagesPullingTime="369.139082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:32:27.628089769 +0000 UTC m=+505.295507961" watchObservedRunningTime="2026-07-07 04:32:27.631291181 +0000 UTC m=+505.298709372" Jul 07 04:32:27.637758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.637730 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zjk2\" (UniqueName: \"kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2\") pod \"441000c3-d6e5-4321-b90e-b374b4973406\" (UID: \"441000c3-d6e5-4321-b90e-b374b4973406\") " Jul 07 04:32:27.639525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.639497 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2" (OuterVolumeSpecName: "kube-api-access-9zjk2") pod "441000c3-d6e5-4321-b90e-b374b4973406" (UID: "441000c3-d6e5-4321-b90e-b374b4973406"). InnerVolumeSpecName "kube-api-access-9zjk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:32:27.738459 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.738428 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjk2\" (UniqueName: \"kubernetes.io/projected/441000c3-d6e5-4321-b90e-b374b4973406-kube-api-access-9zjk2\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:32:27.872971 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.872899 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:27.940121 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.940087 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmk8m\" (UniqueName: \"kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m\") pod \"4973ec21-6e2b-4845-8ed8-c1b26643fbc3\" (UID: \"4973ec21-6e2b-4845-8ed8-c1b26643fbc3\") " Jul 07 04:32:27.942214 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:27.942183 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m" (OuterVolumeSpecName: "kube-api-access-jmk8m") pod "4973ec21-6e2b-4845-8ed8-c1b26643fbc3" (UID: "4973ec21-6e2b-4845-8ed8-c1b26643fbc3"). InnerVolumeSpecName "kube-api-access-jmk8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:32:28.041060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.040963 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmk8m\" (UniqueName: \"kubernetes.io/projected/4973ec21-6e2b-4845-8ed8-c1b26643fbc3-kube-api-access-jmk8m\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:32:28.626538 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.626494 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" event={"ID":"0890db77-3afd-47d4-84e3-615a74adc81d","Type":"ContainerStarted","Data":"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3"} Jul 07 04:32:28.628029 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.627988 2560 generic.go:356] "Generic (PLEG): container finished" podID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" containerID="1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49" exitCode=0 Jul 07 04:32:28.628170 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.628072 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" Jul 07 04:32:28.628170 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.628082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cfcc4fc7b-qqgjk" Jul 07 04:32:28.628170 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.628069 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" event={"ID":"4973ec21-6e2b-4845-8ed8-c1b26643fbc3","Type":"ContainerDied","Data":"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49"} Jul 07 04:32:28.628326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.628178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7jvj9" event={"ID":"4973ec21-6e2b-4845-8ed8-c1b26643fbc3","Type":"ContainerDied","Data":"91c10e7e6e9f36f9b7aa358f580c9c73dee02e1439a746d4b5d4ec46e997f785"} Jul 07 04:32:28.628326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.628195 2560 scope.go:117] "RemoveContainer" containerID="1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49" Jul 07 04:32:28.637362 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.637342 2560 scope.go:117] "RemoveContainer" containerID="1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49" Jul 07 04:32:28.637678 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:32:28.637655 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49\": container with ID starting with 1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49 not found: ID does not exist" containerID="1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49" Jul 07 04:32:28.637740 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.637685 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49"} err="failed to get container status \"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49\": rpc error: code = NotFound desc = could not find container \"1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49\": container with ID starting with 1231f4789d8c2bd33b035f21ce721d074d9a8c97e9b29ebe4e5bad9a9ae88d49 not found: ID does not exist" Jul 07 04:32:28.643327 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.643283 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" podStartSLOduration=2.305719864 podStartE2EDuration="2.643267509s" totalImagesPullingTime="337.547645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:32:28.642700581 +0000 UTC m=+506.310118770" watchObservedRunningTime="2026-07-07 04:32:28.643267509 +0000 UTC m=+506.310685701" Jul 07 04:32:28.669198 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.669164 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cfcc4fc7b-qqgjk"] Jul 07 04:32:28.677691 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.677662 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:32:28.677882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.677857 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-c4dgw" podUID="90dc0569-925c-42a1-8f76-ff0b72d42e84" containerName="authorino" containerID="cri-o://9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc" gracePeriod=30 Jul 07 04:32:28.679681 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.679657 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cfcc4fc7b-qqgjk"] Jul 07 04:32:28.697243 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.697209 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:28.699063 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.699038 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7jvj9"] Jul 07 04:32:28.927327 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.927296 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:32:28.950360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.950326 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tch27\" (UniqueName: \"kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27\") pod \"90dc0569-925c-42a1-8f76-ff0b72d42e84\" (UID: \"90dc0569-925c-42a1-8f76-ff0b72d42e84\") " Jul 07 04:32:28.952659 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:28.952623 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27" (OuterVolumeSpecName: "kube-api-access-tch27") pod "90dc0569-925c-42a1-8f76-ff0b72d42e84" (UID: "90dc0569-925c-42a1-8f76-ff0b72d42e84"). InnerVolumeSpecName "kube-api-access-tch27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:32:29.051948 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.051911 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tch27\" (UniqueName: \"kubernetes.io/projected/90dc0569-925c-42a1-8f76-ff0b72d42e84-kube-api-access-tch27\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:32:29.236263 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.236187 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441000c3-d6e5-4321-b90e-b374b4973406" path="/var/lib/kubelet/pods/441000c3-d6e5-4321-b90e-b374b4973406/volumes" Jul 07 04:32:29.236420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.236409 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" path="/var/lib/kubelet/pods/4973ec21-6e2b-4845-8ed8-c1b26643fbc3/volumes" Jul 07 04:32:29.635322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.635290 2560 generic.go:356] "Generic (PLEG): container finished" podID="90dc0569-925c-42a1-8f76-ff0b72d42e84" containerID="9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc" exitCode=0 Jul 07 04:32:29.635854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.635333 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-c4dgw" Jul 07 04:32:29.635854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.635376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c4dgw" event={"ID":"90dc0569-925c-42a1-8f76-ff0b72d42e84","Type":"ContainerDied","Data":"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc"} Jul 07 04:32:29.635854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.635422 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-c4dgw" event={"ID":"90dc0569-925c-42a1-8f76-ff0b72d42e84","Type":"ContainerDied","Data":"1d39687d79bbd535b799750e22816073acdba61cc8e3dc978c2e095c98f8add2"} Jul 07 04:32:29.635854 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.635446 2560 scope.go:117] "RemoveContainer" containerID="9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc" Jul 07 04:32:29.643882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.643864 2560 scope.go:117] "RemoveContainer" containerID="9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc" Jul 07 04:32:29.644151 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:32:29.644126 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc\": container with ID starting with 9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc not found: ID does not exist" containerID="9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc" Jul 07 04:32:29.644221 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.644161 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc"} err="failed to get container status \"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc\": rpc error: code = NotFound desc = could not find container \"9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc\": container with ID starting with 9606eedd7c36f117c6be6644be4cc8b5cac4aa4a9cd7820f145319bdb142a4fc not found: ID does not exist" Jul 07 04:32:29.653245 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.653218 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:32:29.655722 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:29.655703 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-c4dgw"] Jul 07 04:32:31.236498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:31.236459 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90dc0569-925c-42a1-8f76-ff0b72d42e84" path="/var/lib/kubelet/pods/90dc0569-925c-42a1-8f76-ff0b72d42e84/volumes" Jul 07 04:32:32.995667 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:32.995624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:33.101561 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:33.101518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:33.167148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:33.167111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:33.360068 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:33.360037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:36.140842 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:36.140800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:32:36.235803 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:32:36.235765 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:05.465178 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:05.465144 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:07.650354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:07.650313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:08.186766 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:08.186734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:08.593319 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:08.593287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:08.729929 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:08.729899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:08.833377 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:08.833324 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:08.959341 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:08.959249 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:33:13.312127 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:13.312077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:13.373699 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:13.373659 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:18.097616 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:18.097545 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:18.191733 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:18.191698 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:32.931241 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:32.931203 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:33.337463 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:33.337420 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:33:45.827754 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:45.827720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:33:46.798252 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:33:46.798199 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:34:03.137231 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:03.137206 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:34:03.137874 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:03.137857 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:34:03.806037 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:03.805991 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:34:04.791915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:04.791873 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:34:33.280442 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:33.280405 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:34:34.700499 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:34.700455 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:34:41.806582 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:41.806535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:34:41.976909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:41.976875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:34:43.800996 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:43.800910 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:34:49.305305 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:49.305269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:34:51.105788 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:34:51.105753 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:35:24.174413 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:24.174372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:24.274998 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:24.274957 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:31.337516 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:31.337478 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:36.651635 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.651601 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d95d4cf77-jvj9g"] Jul 07 04:35:36.652041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.651928 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" containerName="authorino" Jul 07 04:35:36.652041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.651939 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" containerName="authorino" Jul 07 04:35:36.652041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.651953 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90dc0569-925c-42a1-8f76-ff0b72d42e84" containerName="authorino" Jul 07 04:35:36.652041 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.651961 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="90dc0569-925c-42a1-8f76-ff0b72d42e84" containerName="authorino" Jul 07 04:35:36.652195 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.652050 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4973ec21-6e2b-4845-8ed8-c1b26643fbc3" containerName="authorino" Jul 07 04:35:36.652195 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.652061 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="90dc0569-925c-42a1-8f76-ff0b72d42e84" containerName="authorino" Jul 07 04:35:36.655084 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.655068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.663867 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.663839 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d95d4cf77-jvj9g"] Jul 07 04:35:36.738064 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.738026 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5643f5ec-ddea-4c27-8aec-3c7677752883-tls-cert\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.738242 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.738085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xjk\" (UniqueName: \"kubernetes.io/projected/5643f5ec-ddea-4c27-8aec-3c7677752883-kube-api-access-b4xjk\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.839364 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.839333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5643f5ec-ddea-4c27-8aec-3c7677752883-tls-cert\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.839466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.839383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xjk\" (UniqueName: \"kubernetes.io/projected/5643f5ec-ddea-4c27-8aec-3c7677752883-kube-api-access-b4xjk\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.841743 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.841714 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5643f5ec-ddea-4c27-8aec-3c7677752883-tls-cert\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.847125 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.847102 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xjk\" (UniqueName: \"kubernetes.io/projected/5643f5ec-ddea-4c27-8aec-3c7677752883-kube-api-access-b4xjk\") pod \"authorino-7d95d4cf77-jvj9g\" (UID: \"5643f5ec-ddea-4c27-8aec-3c7677752883\") " pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:36.964470 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:36.964383 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" Jul 07 04:35:37.097880 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:37.097854 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d95d4cf77-jvj9g"] Jul 07 04:35:37.100238 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:35:37.100214 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5643f5ec_ddea_4c27_8aec_3c7677752883.slice/crio-7f27a7dfe5bea2c7c49813d04142b79880730957829fc575058ac17551c03128 WatchSource:0}: Error finding container 7f27a7dfe5bea2c7c49813d04142b79880730957829fc575058ac17551c03128: Status 404 returned error can't find the container with id 7f27a7dfe5bea2c7c49813d04142b79880730957829fc575058ac17551c03128 Jul 07 04:35:37.101543 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:37.101523 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:35:37.556481 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:37.556444 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" event={"ID":"5643f5ec-ddea-4c27-8aec-3c7677752883","Type":"ContainerStarted","Data":"7f27a7dfe5bea2c7c49813d04142b79880730957829fc575058ac17551c03128"} Jul 07 04:35:38.564594 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:38.564550 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" event={"ID":"5643f5ec-ddea-4c27-8aec-3c7677752883","Type":"ContainerStarted","Data":"e6d541ce135f71c5de72d5175fc5314ce3725e60ef4880869cb6573fd5dfdec8"} Jul 07 04:35:38.585250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:38.585183 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d95d4cf77-jvj9g" podStartSLOduration=2.010885976 podStartE2EDuration="2.585159809s" totalImagesPullingTime="574.273833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:35:38.58133032 +0000 UTC m=+696.248748514" watchObservedRunningTime="2026-07-07 04:35:38.585159809 +0000 UTC m=+696.252578002" Jul 07 04:35:38.621542 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:38.621505 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:35:38.621742 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:38.621710 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" podUID="0890db77-3afd-47d4-84e3-615a74adc81d" containerName="authorino" containerID="cri-o://cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3" gracePeriod=30 Jul 07 04:35:38.890705 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:38.890677 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:35:39.062047 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.061985 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qmfq\" (UniqueName: \"kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq\") pod \"0890db77-3afd-47d4-84e3-615a74adc81d\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " Jul 07 04:35:39.062217 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.062064 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert\") pod \"0890db77-3afd-47d4-84e3-615a74adc81d\" (UID: \"0890db77-3afd-47d4-84e3-615a74adc81d\") " Jul 07 04:35:39.064168 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.064131 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq" (OuterVolumeSpecName: "kube-api-access-5qmfq") pod "0890db77-3afd-47d4-84e3-615a74adc81d" (UID: "0890db77-3afd-47d4-84e3-615a74adc81d"). InnerVolumeSpecName "kube-api-access-5qmfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:35:39.072314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.072292 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0890db77-3afd-47d4-84e3-615a74adc81d" (UID: "0890db77-3afd-47d4-84e3-615a74adc81d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:35:39.135503 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.135412 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:39.163245 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.163207 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmfq\" (UniqueName: \"kubernetes.io/projected/0890db77-3afd-47d4-84e3-615a74adc81d-kube-api-access-5qmfq\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:35:39.163245 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.163236 2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0890db77-3afd-47d4-84e3-615a74adc81d-tls-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:35:39.226440 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.226404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:39.571347 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.571312 2560 generic.go:356] "Generic (PLEG): container finished" podID="0890db77-3afd-47d4-84e3-615a74adc81d" containerID="cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3" exitCode=0 Jul 07 04:35:39.571823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.571366 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" Jul 07 04:35:39.571823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.571362 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" event={"ID":"0890db77-3afd-47d4-84e3-615a74adc81d","Type":"ContainerDied","Data":"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3"} Jul 07 04:35:39.571823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.571485 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfd88c964-8hqjf" event={"ID":"0890db77-3afd-47d4-84e3-615a74adc81d","Type":"ContainerDied","Data":"18c84d2da7b8349c9ece595c4e9b6bc5a6ba0c43a831654b9e482d3f606f8d86"} Jul 07 04:35:39.571823 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.571503 2560 scope.go:117] "RemoveContainer" containerID="cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3" Jul 07 04:35:39.580702 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.580684 2560 scope.go:117] "RemoveContainer" containerID="cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3" Jul 07 04:35:39.580971 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:35:39.580951 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3\": container with ID starting with cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3 not found: ID does not exist" containerID="cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3" Jul 07 04:35:39.581042 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.580981 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3"} err="failed to get container status \"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3\": rpc error: code = NotFound desc = could not find container \"cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3\": container with ID starting with cb88fdc31aff7a7e4d2b626ca02b5331b9afe43cce73c7ff94edbef45be4d9e3 not found: ID does not exist" Jul 07 04:35:39.588208 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.588180 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:35:39.596256 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:39.596234 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cfd88c964-8hqjf"] Jul 07 04:35:40.318235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:40.318202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:40.409029 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:40.408973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:41.235377 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:41.235348 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0890db77-3afd-47d4-84e3-615a74adc81d" path="/var/lib/kubelet/pods/0890db77-3afd-47d4-84e3-615a74adc81d/volumes" Jul 07 04:35:44.154405 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:44.154372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:35:59.259522 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:35:59.259435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:36:08.501806 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:08.501768 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:36:19.110025 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:19.109967 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:36:29.020374 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:29.020337 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:36:39.427872 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:39.427829 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:36:49.394915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:49.394877 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:36:51.696443 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:51.696403 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:36:53.076314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:36:53.076283 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:16.503955 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:16.503917 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:37:27.900601 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:27.900519 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:37:33.481305 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:33.481247 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:33.580266 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:33.580229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:33.694961 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:33.694925 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:39.532216 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:39.532181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:39.597511 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:39.597480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:39.670901 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:39.670865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:39.771016 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:39.770972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:39.896915 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:39.896881 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:45.691530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:45.691496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:37:45.791665 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:37:45.791630 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:38:03.292876 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:03.292842 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:12.004664 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:12.004625 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:20.305938 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:20.305905 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:30.392847 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:30.392812 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:38.713908 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:38.713863 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:47.142487 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:47.142452 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:38:59.305250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:38:59.305170 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:39:03.170968 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:39:03.170940 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:39:03.171453 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:39:03.171291 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:39:08.308217 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:39:08.308164 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:39:31.398067 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:39:31.398030 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:39:40.295093 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:39:40.295053 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:40:04.003360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:40:04.003315 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:40:15.604865 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:40:15.604825 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:40:30.527110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:40:30.510726 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:40:40.309704 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:40:40.309662 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:40:50.911031 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:40:50.910986 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:41:00.306790 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:41:00.306752 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:41:07.820971 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:41:07.820926 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:41:25.477694 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:41:25.477654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:41:26.570250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:41:26.570213 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:41:28.998377 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:41:28.998333 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:42:08.980738 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:42:08.980646 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:42:57.400993 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:42:57.400950 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:00.422806 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:00.418618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:43:08.007224 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:08.007191 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:16.704857 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:16.704821 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:27.004546 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:27.004448 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:37.507058 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:37.507018 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:47.804078 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:47.804036 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:43:55.099595 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:43:55.099543 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:44:03.205908 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:03.205878 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:44:03.208707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:03.208686 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:44:04.397036 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:04.396988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:44:13.103096 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:13.103059 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:44:23.803671 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:23.803633 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:44:34.800459 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:34.800425 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:44:42.999305 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:44:42.999260 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:03.396108 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:03.396034 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:10.505312 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:10.505226 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:20.305033 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:20.304985 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:30.334289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:30.334251 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:39.295525 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:39.295495 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:45:51.598979 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:45:51.598945 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:05.501531 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:05.501498 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:14.001663 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:14.001624 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:19.701417 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:19.701386 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:28.913559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:28.913466 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:37.700515 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:37.700480 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:50.304499 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:50.304464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:46:59.008166 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:46:59.008135 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:47:09.105415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:09.105372 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:47:11.995293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:11.995259 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:47:19.296866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:19.296827 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:47:29.404636 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:29.404599 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:47:37.206285 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:37.206238 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:47:48.903819 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:47:48.903784 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:00.904544 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:00.904459 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:11.400732 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:11.400697 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:16.273310 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:16.273266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:20.536156 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:20.536119 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:24.103115 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:24.103078 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:28.842323 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:28.842293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:32.602555 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:32.602514 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:33.987749 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:33.987700 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:37.071391 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:37.071355 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:37.169559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:37.169510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:37.545339 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:37.545234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:42.102218 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:42.102166 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:48:43.518307 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:43.518266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:43.582564 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:43.582529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:43.743378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:43.743346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:43.817498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:43.817464 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:44.856755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:44.856724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:44.947386 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:44.947348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:45.014972 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:45.014939 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:45.115435 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:45.115347 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:45.404524 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:45.404439 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:45.502240 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:45.502208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:47.258695 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:47.258665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:50.032315 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:50.032287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:50.316928 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:50.316897 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:50.407268 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:50.407235 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:50.524557 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:50.524521 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:51.727213 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:51.727186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:56.046917 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:56.046863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:56.113779 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:56.113747 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:56.272604 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:56.272572 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:56.334050 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:56.334016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:57.409191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:57.409159 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:57.487286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:57.487256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:57.547313 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:57.547283 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:57.772304 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:57.772211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:58.013615 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:58.013579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:58.115984 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:58.115951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:59.804631 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:59.804597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:48:59.998916 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:48:59.998884 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:49:02.746140 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:02.746105 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:03.078773 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:03.078738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:03.239100 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:03.239076 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:49:03.242902 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:03.242876 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:49:08.937061 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:08.937022 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:09.037183 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:09.037091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:10.312116 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:10.312079 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:10.413348 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:10.413310 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:11.292024 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:11.291972 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:49:14.791664 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:14.791620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:14.893474 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:14.893444 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:14.993397 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:14.993350 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.102168 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.102130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.195261 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.195231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.282804 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.282769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.475312 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.475272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.572680 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.572651 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.783883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.783801 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:16.882177 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:16.882140 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:19.966197 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:19.966166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:20.591205 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:20.591169 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:20.805201 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:20.805137 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:20.922237 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:20.922124 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:20.980919 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:20.980890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:22.740142 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:22.740108 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:24.052106 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:24.052072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:24.154928 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:24.154872 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:26.757454 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:26.757344 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:26.824952 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:26.824905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:27.299425 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:27.299350 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:27.400578 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:27.400543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:27.567250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:27.567213 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:29.004883 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:29.004847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:34.637568 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:34.637536 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:34.739359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:34.739302 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:34.836174 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:34.836144 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:34.936158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:34.936078 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:49:35.398947 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:49:35.398916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:01.325752 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:01.325720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:06.935977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:06.935943 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:07.003617 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:07.003564 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:07.083326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:07.083299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:07.181718 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:07.181690 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:23.108142 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:23.108108 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:28.970631 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:28.970597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:29.079789 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:29.079756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:29.174997 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:29.174972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:39.705332 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:39.705295 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:40.485911 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:40.485876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:46.204244 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:46.204187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:46.304807 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:46.304773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:46.401755 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:46.401721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:46.501383 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:46.501290 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:46.643259 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:46.643227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:47.288944 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:47.288904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:47.387688 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:47.387656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:47.489762 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:47.489729 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:49.007813 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:49.007777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:50:49.066103 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:49.066069 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:54.891245 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:54.891164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:54.955196 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:54.955154 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:55.560191 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:55.560157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:50:57.598345 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:50:57.598305 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:51:01.491601 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:01.491569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:01.592281 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:01.592250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:01.723219 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:01.723186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:02.049148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:02.049122 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:02.225251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:02.225220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.208190 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.208146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.305075 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.305044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.370129 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.370098 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.776300 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.776269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.877110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.877084 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:05.988858 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:05.988823 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:09.400365 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:09.400333 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:51:09.729958 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:09.729887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:10.377710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:10.377679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:10.476576 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:10.476538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:10.802913 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:10.802757 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:16.652532 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:16.652495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:16.717669 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:16.717638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:17.147394 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:17.147353 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:17.517259 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:17.517176 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:17.577622 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:17.577600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:22.203065 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:22.203033 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:51:23.440758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:23.440731 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:23.513742 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:23.513715 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:23.701792 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:23.701709 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:23.933185 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:23.933147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:24.045873 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:24.045795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:24.142203 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:24.142168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:24.261920 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:24.261885 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:30.041420 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:30.041382 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:30.108847 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:30.108810 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:30.180680 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:30.180645 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:30.280092 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:30.280058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:30.396524 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:30.396491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:36.145470 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:36.145434 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:36.246620 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:36.246588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:42.942485 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:42.942450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:43.262599 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:43.262523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:43.524680 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:43.524585 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:46.454920 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:46.454888 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:46.542863 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:46.542821 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:46.607193 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:46.607164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:46.927488 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:46.927453 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:47.120118 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:47.120087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:47.283740 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:47.283636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:50.938991 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:50.938960 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:51:52.599994 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:51:52.599955 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:52:12.998368 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:12.998337 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:52:12.998739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:12.998668 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0890db77-3afd-47d4-84e3-615a74adc81d" containerName="authorino" Jul 07 04:52:12.998739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:12.998678 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0890db77-3afd-47d4-84e3-615a74adc81d" containerName="authorino" Jul 07 04:52:12.998739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:12.998733 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0890db77-3afd-47d4-84e3-615a74adc81d" containerName="authorino" Jul 07 04:52:13.001321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.001305 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.005274 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.005241 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-036fb253\"/\"default-dockercfg-c9dkv\"" Jul 07 04:52:13.005401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.005314 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-036fb253\"/\"e2e-lifecycle-model-036fb253-kserve-self-signed-certs\"" Jul 07 04:52:13.005401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.005363 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-036fb253\"/\"kube-root-ca.crt\"" Jul 07 04:52:13.005509 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.005450 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-036fb253\"/\"openshift-service-ca.crt\"" Jul 07 04:52:13.010326 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.010306 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:52:13.017652 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.017628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:13.088580 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088552 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.088580 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.088783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.088783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088660 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wq6\" (UniqueName: \"kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.088783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088696 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.088783 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.088759 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wq6\" (UniqueName: \"kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190262 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190257 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190530 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190506 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190569 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190543 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.190612 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.190597 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.192460 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.192435 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.192605 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.192590 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.198060 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.198038 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wq6\" (UniqueName: \"kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6\") pod \"e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.310671 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.310578 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:13.446972 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.446948 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:52:13.448872 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:52:13.448848 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod768bf54d_c9a1_4bc7_bb8a_d150595765e4.slice/crio-791088c512ca2bb65f894949c2d360bee78063b806e0570f694e886d0a798411 WatchSource:0}: Error finding container 791088c512ca2bb65f894949c2d360bee78063b806e0570f694e886d0a798411: Status 404 returned error can't find the container with id 791088c512ca2bb65f894949c2d360bee78063b806e0570f694e886d0a798411 Jul 07 04:52:13.451280 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.451261 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 04:52:13.495673 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:13.495635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" event={"ID":"768bf54d-c9a1-4bc7-bb8a-d150595765e4","Type":"ContainerStarted","Data":"791088c512ca2bb65f894949c2d360bee78063b806e0570f694e886d0a798411"} Jul 07 04:52:15.507125 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:15.507085 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" event={"ID":"768bf54d-c9a1-4bc7-bb8a-d150595765e4","Type":"ContainerStarted","Data":"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c"} Jul 07 04:52:15.526844 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:15.526798 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" podStartSLOduration=2.43831519 podStartE2EDuration="3.526784706s" totalImagesPullingTime="1.088469516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:52:15.524474426 +0000 UTC m=+1693.191892616" watchObservedRunningTime="2026-07-07 04:52:15.526784706 +0000 UTC m=+1693.194202897" Jul 07 04:52:15.978659 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:15.978623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:16.300137 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:16.300055 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:52:17.005707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:17.005673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:23.311344 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:23.311303 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:23.311739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:23.311362 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:23.323526 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:23.323499 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:23.556321 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:23.556292 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:52:32.999642 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:32.999552 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:52:38.689400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:38.689364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:39.123847 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:39.123812 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:39.224444 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:39.224388 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:39.377540 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:39.377451 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:42.299494 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:42.299460 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:52:44.788077 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:44.788040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:49.768342 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:49.768307 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:52:49.864554 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:49.864522 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:52:54.298171 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:52:54.298135 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:53:04.788352 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:04.788308 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" podUID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" containerName="main" containerID="cri-o://67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c" gracePeriod=45 Jul 07 04:53:05.032016 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.031987 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:53:05.112792 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.112977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112806 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.112977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112826 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.112977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112870 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6wq6\" (UniqueName: \"kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.112977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112887 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.112977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.112909 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache\") pod \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\" (UID: \"768bf54d-c9a1-4bc7-bb8a-d150595765e4\") " Jul 07 04:53:05.113258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.113093 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home" (OuterVolumeSpecName: "home") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:53:05.113258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.113192 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:53:05.113258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.113208 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.113258 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.113225 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:53:05.115016 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.114978 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm" (OuterVolumeSpecName: "dshm") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:53:05.115016 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.114981 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:53:05.115131 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.114981 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6" (OuterVolumeSpecName: "kube-api-access-k6wq6") pod "768bf54d-c9a1-4bc7-bb8a-d150595765e4" (UID: "768bf54d-c9a1-4bc7-bb8a-d150595765e4"). InnerVolumeSpecName "kube-api-access-k6wq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:53:05.214094 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.214062 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.214094 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.214087 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.214094 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.214098 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6wq6\" (UniqueName: \"kubernetes.io/projected/768bf54d-c9a1-4bc7-bb8a-d150595765e4-kube-api-access-k6wq6\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.214314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.214106 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768bf54d-c9a1-4bc7-bb8a-d150595765e4-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.214314 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.214116 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768bf54d-c9a1-4bc7-bb8a-d150595765e4-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:53:05.755861 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.755828 2560 generic.go:356] "Generic (PLEG): container finished" podID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" containerID="67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c" exitCode=0 Jul 07 04:53:05.756090 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.755877 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" Jul 07 04:53:05.756090 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.755922 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" event={"ID":"768bf54d-c9a1-4bc7-bb8a-d150595765e4","Type":"ContainerDied","Data":"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c"} Jul 07 04:53:05.756090 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.755967 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk" event={"ID":"768bf54d-c9a1-4bc7-bb8a-d150595765e4","Type":"ContainerDied","Data":"791088c512ca2bb65f894949c2d360bee78063b806e0570f694e886d0a798411"} Jul 07 04:53:05.756090 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.755990 2560 scope.go:117] "RemoveContainer" containerID="67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c" Jul 07 04:53:05.764336 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.764321 2560 scope.go:117] "RemoveContainer" containerID="67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c" Jul 07 04:53:05.764562 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:53:05.764546 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c\": container with ID starting with 67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c not found: ID does not exist" containerID="67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c" Jul 07 04:53:05.764625 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.764568 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c"} err="failed to get container status \"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c\": rpc error: code = NotFound desc = could not find container \"67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c\": container with ID starting with 67d99c34788f700a47645e4a4f7d7b5a941e6769d6b44b206fb680c2bcad1c8c not found: ID does not exist" Jul 07 04:53:05.773569 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.773543 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:53:05.777465 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:05.777435 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-036fb253/e2e-lifecycle-model-036fb253-kserve-d8777cbff-gc5bk"] Jul 07 04:53:07.235990 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:07.235956 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" path="/var/lib/kubelet/pods/768bf54d-c9a1-4bc7-bb8a-d150595765e4/volumes" Jul 07 04:53:25.935918 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:25.935887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:27.023286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:27.023256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:29.100184 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:29.100149 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:53:32.733080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:32.733047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:32.840380 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:32.840352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:33.012933 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:33.012854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:33.243905 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:33.243878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:34.312070 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:34.312036 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:34.414121 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:34.414056 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:34.515399 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:34.515335 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:35.213030 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:35.212977 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:35.311487 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:35.311450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:35.411849 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:35.411798 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:36.759279 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:36.759243 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:42.496101 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:42.496070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:42.551035 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:42.550987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:43.181353 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:43.181323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:43.407301 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:43.407274 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:53:48.975023 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:48.974973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:49.088167 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:49.088135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:49.204423 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:49.204392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:53:51.098426 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:51.098382 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:53:59.678231 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:53:59.678146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:00.024469 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:00.024389 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:03.270250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:03.270222 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:54:03.279523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:03.279499 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:54:20.972980 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:20.972932 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:26.839299 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:26.839269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:26.904299 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:26.904271 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:27.214112 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:27.214040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:27.518202 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:27.518116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:27.739463 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:27.739430 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:30.352097 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:30.352064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:30.453397 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:30.453368 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:30.552483 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:30.552447 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:31.088519 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:31.088490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:31.187631 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:31.187599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:31.292040 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:31.291996 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:34.757071 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:34.757042 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:35.076731 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:35.076692 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:35.409360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:35.409291 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:36.655947 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:36.655913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:38.091578 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.091548 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:54:38.305196 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.305164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:38.389474 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.389408 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:38.456634 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.456604 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:38.629512 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.629479 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:38.959739 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:38.959708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:39.060324 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:39.060291 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:42.818347 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:42.818309 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:51.812288 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:51.812240 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:51.911349 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:51.911313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:52.000366 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:52.000331 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:54:54.499814 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.499778 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:54:54.500306 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.500287 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" containerName="main" Jul 07 04:54:54.500360 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.500310 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" containerName="main" Jul 07 04:54:54.500400 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.500388 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="768bf54d-c9a1-4bc7-bb8a-d150595765e4" containerName="main" Jul 07 04:54:54.503615 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.503595 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.507448 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.507412 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-94ebac\"/\"kube-root-ca.crt\"" Jul 07 04:54:54.507808 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.507786 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-94ebac\"/\"test-model-94ebac-kserve-self-signed-certs\"" Jul 07 04:54:54.507909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.507786 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-94ebac\"/\"default-dockercfg-8ntn7\"" Jul 07 04:54:54.507909 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.507783 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-94ebac\"/\"openshift-service-ca.crt\"" Jul 07 04:54:54.510650 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.510627 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:54:54.584989 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.584960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.585158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff88\" (UniqueName: \"kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.585158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.585158 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585114 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.585293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.585293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:54.585293 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.585244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686626 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686626 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686840 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686840 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686682 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686840 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686714 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.686840 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.686739 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff88\" (UniqueName: \"kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.687080 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.687056 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.687143 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.687081 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.687201 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.687183 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.688955 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.688934 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.689148 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.689132 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.697224 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.697203 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff88\" (UniqueName: \"kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88\") pod \"test-model-94ebac-kserve-67f75cb69c-8hz4v\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.814350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.814325 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:54:54.938466 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:54.938433 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:54:54.940389 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:54:54.940364 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda12cf392_a447_47be_b2e2_304e903a39d2.slice/crio-7b2e8f9cd28cbdd8f77df78e6cbe040a9318e2034498ed1f611f7b86639b4791 WatchSource:0}: Error finding container 7b2e8f9cd28cbdd8f77df78e6cbe040a9318e2034498ed1f611f7b86639b4791: Status 404 returned error can't find the container with id 7b2e8f9cd28cbdd8f77df78e6cbe040a9318e2034498ed1f611f7b86639b4791 Jul 07 04:54:55.292705 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:55.292613 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" event={"ID":"a12cf392-a447-47be-b2e2-304e903a39d2","Type":"ContainerStarted","Data":"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96"} Jul 07 04:54:55.292705 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:55.292655 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" event={"ID":"a12cf392-a447-47be-b2e2-304e903a39d2","Type":"ContainerStarted","Data":"7b2e8f9cd28cbdd8f77df78e6cbe040a9318e2034498ed1f611f7b86639b4791"} Jul 07 04:54:55.310758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:55.310713 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" podStartSLOduration=1.310695634 podStartE2EDuration="1.310695634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:54:55.308479064 +0000 UTC m=+1852.975897255" watchObservedRunningTime="2026-07-07 04:54:55.310695634 +0000 UTC m=+1852.978113826" Jul 07 04:54:59.803827 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:59.803795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:54:59.900709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:54:59.900676 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:02.004715 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:02.004677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:55:04.814559 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:04.814524 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:04.814988 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:04.814589 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:04.827245 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:04.827224 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:05.353073 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:05.353044 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:18.483317 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.483279 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:18.486614 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.486598 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.490384 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.490361 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee0733\"/\"kube-root-ca.crt\"" Jul 07 04:55:18.490498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.490400 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee0733\"/\"openshift-service-ca.crt\"" Jul 07 04:55:18.490498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.490429 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee0733\"/\"test-model-ee0733-kserve-self-signed-certs\"" Jul 07 04:55:18.490498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.490402 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee0733\"/\"default-dockercfg-hmfqf\"" Jul 07 04:55:18.493931 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.493908 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:18.508828 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.508804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:18.573535 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2rg\" (UniqueName: \"kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.573632 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.573632 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573566 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.573704 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.573704 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573646 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.573704 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.573674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.674707 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674682 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2rg\" (UniqueName: \"kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.674817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674714 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.674817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674734 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.674817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.674817 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.675030 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.674825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.675207 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.675181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.675282 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.675239 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.675282 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.675267 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.677107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.677083 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.677384 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.677365 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.682961 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.682937 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2rg\" (UniqueName: \"kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg\") pod \"test-model-ee0733-kserve-7d9b9977f5-hdwcn\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.796081 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.796019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:18.920568 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:18.920539 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:18.922306 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:55:18.922281 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0671efba_6bda_4c8d_88c4_58a4edf53e7c.slice/crio-0594f3d7fa1ec3b9fa6eb1b341c69583bec2a4a8178b44dbd5019cbb807041be WatchSource:0}: Error finding container 0594f3d7fa1ec3b9fa6eb1b341c69583bec2a4a8178b44dbd5019cbb807041be: Status 404 returned error can't find the container with id 0594f3d7fa1ec3b9fa6eb1b341c69583bec2a4a8178b44dbd5019cbb807041be Jul 07 04:55:19.409498 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:19.409460 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" event={"ID":"0671efba-6bda-4c8d-88c4-58a4edf53e7c","Type":"ContainerStarted","Data":"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d"} Jul 07 04:55:19.409678 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:19.409504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" event={"ID":"0671efba-6bda-4c8d-88c4-58a4edf53e7c","Type":"ContainerStarted","Data":"0594f3d7fa1ec3b9fa6eb1b341c69583bec2a4a8178b44dbd5019cbb807041be"} Jul 07 04:55:19.427165 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:19.427120 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" podStartSLOduration=1.42710524 podStartE2EDuration="1.42710524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:55:19.424344215 +0000 UTC m=+1877.091762407" watchObservedRunningTime="2026-07-07 04:55:19.42710524 +0000 UTC m=+1877.094523454" Jul 07 04:55:24.401345 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:24.401245 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:55:28.796354 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:28.796316 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:28.796771 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:28.796368 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:28.808583 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:28.808560 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:29.469161 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:29.469127 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:36.280746 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:36.280711 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:55:36.289232 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:36.289204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:37.148238 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:37.148202 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:37.160536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:37.160511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:37.342930 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:37.342902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:37.820287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:37.820257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:37.922694 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:37.922662 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:38.022813 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:38.022780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:38.127308 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:38.127232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:41.499366 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:41.499331 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:55:42.899393 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:42.899362 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:51.300729 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.300681 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" podUID="a12cf392-a447-47be-b2e2-304e903a39d2" containerName="main" containerID="cri-o://51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96" gracePeriod=45 Jul 07 04:55:51.550892 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.550837 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:51.571565 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.571536 2560 generic.go:356] "Generic (PLEG): container finished" podID="a12cf392-a447-47be-b2e2-304e903a39d2" containerID="51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96" exitCode=0 Jul 07 04:55:51.571709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.571581 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" Jul 07 04:55:51.571709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.571633 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" event={"ID":"a12cf392-a447-47be-b2e2-304e903a39d2","Type":"ContainerDied","Data":"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96"} Jul 07 04:55:51.571709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.571677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v" event={"ID":"a12cf392-a447-47be-b2e2-304e903a39d2","Type":"ContainerDied","Data":"7b2e8f9cd28cbdd8f77df78e6cbe040a9318e2034498ed1f611f7b86639b4791"} Jul 07 04:55:51.571709 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.571704 2560 scope.go:117] "RemoveContainer" containerID="51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96" Jul 07 04:55:51.580095 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.580072 2560 scope.go:117] "RemoveContainer" containerID="51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96" Jul 07 04:55:51.580399 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:55:51.580379 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96\": container with ID starting with 51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96 not found: ID does not exist" containerID="51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96" Jul 07 04:55:51.580465 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.580409 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96"} err="failed to get container status \"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96\": rpc error: code = NotFound desc = could not find container \"51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96\": container with ID starting with 51bb6ad5a8c2162d90a28e02a951063709fa91af72198f5c242fe7847d03bc96 not found: ID does not exist" Jul 07 04:55:51.651886 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.651859 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.651986 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.651891 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.651986 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.651908 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.651986 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.651946 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.652147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652070 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.652147 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652117 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vff88\" (UniqueName: \"kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88\") pod \"a12cf392-a447-47be-b2e2-304e903a39d2\" (UID: \"a12cf392-a447-47be-b2e2-304e903a39d2\") " Jul 07 04:55:51.652269 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652181 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:51.652269 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652193 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home" (OuterVolumeSpecName: "home") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:51.652378 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652341 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:51.652464 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652446 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.652501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652471 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.652501 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.652486 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.654095 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.654066 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm" (OuterVolumeSpecName: "dshm") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:51.654189 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.654132 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:55:51.654235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.654186 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88" (OuterVolumeSpecName: "kube-api-access-vff88") pod "a12cf392-a447-47be-b2e2-304e903a39d2" (UID: "a12cf392-a447-47be-b2e2-304e903a39d2"). InnerVolumeSpecName "kube-api-access-vff88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:55:51.753413 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.753392 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a12cf392-a447-47be-b2e2-304e903a39d2-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.753413 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.753412 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a12cf392-a447-47be-b2e2-304e903a39d2-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.753547 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.753421 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff88\" (UniqueName: \"kubernetes.io/projected/a12cf392-a447-47be-b2e2-304e903a39d2-kube-api-access-vff88\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:51.805336 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.805279 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:55:51.893382 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.893356 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:55:51.897562 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:51.897540 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-94ebac/test-model-94ebac-kserve-67f75cb69c-8hz4v"] Jul 07 04:55:52.168388 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.168349 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" podUID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" containerName="main" containerID="cri-o://03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d" gracePeriod=45 Jul 07 04:55:52.417476 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.417457 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:52.560094 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560000 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560094 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560078 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh2rg\" (UniqueName: \"kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560118 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560134 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560182 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560197 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm\") pod \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\" (UID: \"0671efba-6bda-4c8d-88c4-58a4edf53e7c\") " Jul 07 04:55:52.560287 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560264 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:52.560479 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560357 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home" (OuterVolumeSpecName: "home") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:52.560523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560470 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:52.560523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560484 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:52.560523 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.560511 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:52.562298 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.562277 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 04:55:52.562298 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.562292 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm" (OuterVolumeSpecName: "dshm") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 04:55:52.562418 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.562338 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg" (OuterVolumeSpecName: "kube-api-access-vh2rg") pod "0671efba-6bda-4c8d-88c4-58a4edf53e7c" (UID: "0671efba-6bda-4c8d-88c4-58a4edf53e7c"). InnerVolumeSpecName "kube-api-access-vh2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 04:55:52.579322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.579295 2560 generic.go:356] "Generic (PLEG): container finished" podID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" containerID="03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d" exitCode=0 Jul 07 04:55:52.579415 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.579343 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" Jul 07 04:55:52.579464 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.579338 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" event={"ID":"0671efba-6bda-4c8d-88c4-58a4edf53e7c","Type":"ContainerDied","Data":"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d"} Jul 07 04:55:52.579464 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.579446 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn" event={"ID":"0671efba-6bda-4c8d-88c4-58a4edf53e7c","Type":"ContainerDied","Data":"0594f3d7fa1ec3b9fa6eb1b341c69583bec2a4a8178b44dbd5019cbb807041be"} Jul 07 04:55:52.579464 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.579463 2560 scope.go:117] "RemoveContainer" containerID="03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d" Jul 07 04:55:52.587704 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.587687 2560 scope.go:117] "RemoveContainer" containerID="03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d" Jul 07 04:55:52.587943 ip-10-0-142-139 kubenswrapper[2560]: E0707 04:55:52.587926 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d\": container with ID starting with 03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d not found: ID does not exist" containerID="03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d" Jul 07 04:55:52.588031 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.587951 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d"} err="failed to get container status \"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d\": rpc error: code = NotFound desc = could not find container \"03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d\": container with ID starting with 03c1f99b2ebf58a5e2fd1fd940d6673ba65706fed3eba74514f052dba5e5fb3d not found: ID does not exist" Jul 07 04:55:52.601246 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.601222 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:52.606867 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.606844 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ee0733/test-model-ee0733-kserve-7d9b9977f5-hdwcn"] Jul 07 04:55:52.661685 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.661658 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:52.661685 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.661680 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0671efba-6bda-4c8d-88c4-58a4edf53e7c-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:52.661813 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.661693 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2rg\" (UniqueName: \"kubernetes.io/projected/0671efba-6bda-4c8d-88c4-58a4edf53e7c-kube-api-access-vh2rg\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:52.661813 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:52.661706 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0671efba-6bda-4c8d-88c4-58a4edf53e7c-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 04:55:53.236217 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:53.236182 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" path="/var/lib/kubelet/pods/0671efba-6bda-4c8d-88c4-58a4edf53e7c/volumes" Jul 07 04:55:53.236641 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:55:53.236620 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12cf392-a447-47be-b2e2-304e903a39d2" path="/var/lib/kubelet/pods/a12cf392-a447-47be-b2e2-304e903a39d2/volumes" Jul 07 04:56:19.670172 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:19.670102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:19.930228 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:19.930147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:20.159289 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:20.159251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:20.325455 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:20.325427 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:20.735059 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:20.734954 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:20.842228 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:20.842195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:20.986107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:20.986032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:24.090650 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:24.090615 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:56:26.658334 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:26.658302 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:26.758846 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:26.758812 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:26.945330 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:26.945204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:27.185240 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:27.185205 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:27.526978 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:27.526947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:27.840579 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:27.840533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:27.950276 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:27.950241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:30.872187 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:30.872156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:30.974674 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:30.974638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:31.034929 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:31.034898 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:31.342842 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:31.342803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:31.502250 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:31.502222 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:31.601833 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:31.601740 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:35.341481 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:35.341443 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:35.804242 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:35.804148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:35.906070 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:35.906039 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:56:36.119100 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:36.119061 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:36.329327 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:36.329294 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:38.912109 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:38.912076 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:39.004347 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:39.004309 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:39.092732 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:39.092700 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:39.291176 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:39.291081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:39.591933 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:39.591900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:39.699882 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:39.699855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:43.409797 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:43.409764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:47.730065 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:47.730026 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:56:51.475484 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:51.475450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:51.858097 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:51.858061 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:51.959423 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:51.959383 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:52.058745 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:52.058706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:52.160140 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:52.160046 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:57.463692 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:57.463611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:57.520174 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:57.520134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:57.832458 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:57.832424 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:57.967977 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:57.967939 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:58.116993 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:58.116903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:58.517359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:58.517267 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:58.617701 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:58.617658 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:58.718890 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:58.718855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:56:58.826178 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:56:58.826145 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:04.347758 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:04.347727 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:04.449330 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:04.449292 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:04.610235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:04.610142 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:04.796547 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:04.796505 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:57:04.831107 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:04.831071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:05.171340 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:05.171264 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:05.474399 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:05.474311 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:06.110517 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:06.110477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.066099 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.066067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.144056 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.144025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.207430 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.207396 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.310129 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.310091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.622286 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.622250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:08.723794 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:08.723767 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:12.570619 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:12.570590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:12.901969 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:12.901878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:13.207561 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:13.207491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:13.497669 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:13.497573 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:15.205610 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:15.205579 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:57:16.124938 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:16.124903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:16.204957 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:16.204919 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:16.285132 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:16.285100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:16.815126 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:16.815091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:16.915837 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:16.915807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:17.619969 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:17.619936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:19.074994 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:19.074961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:20.640044 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:20.637644 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:29.010866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.010833 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 04:57:29.011251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011193 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" containerName="main" Jul 07 04:57:29.011251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011205 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" containerName="main" Jul 07 04:57:29.011251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011218 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12cf392-a447-47be-b2e2-304e903a39d2" containerName="main" Jul 07 04:57:29.011251 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011223 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12cf392-a447-47be-b2e2-304e903a39d2" containerName="main" Jul 07 04:57:29.011385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011286 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12cf392-a447-47be-b2e2-304e903a39d2" containerName="main" Jul 07 04:57:29.011385 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.011294 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0671efba-6bda-4c8d-88c4-58a4edf53e7c" containerName="main" Jul 07 04:57:29.014197 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.014182 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.018188 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.018167 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-41807c\"/\"auth-test-model-41807c-kserve-self-signed-certs\"" Jul 07 04:57:29.018188 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.018167 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-41807c\"/\"default-dockercfg-lvbv2\"" Jul 07 04:57:29.018350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.018210 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-41807c\"/\"kube-root-ca.crt\"" Jul 07 04:57:29.018350 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.018214 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-41807c\"/\"openshift-service-ca.crt\"" Jul 07 04:57:29.025294 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.025270 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 04:57:29.026235 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.026210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:29.129434 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129405 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.129536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129440 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.129536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.129536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.129536 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8w2h\" (UniqueName: \"kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.129710 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.129548 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230520 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230618 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230724 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230786 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8w2h\" (UniqueName: \"kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230786 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230906 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230889 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.230965 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.230942 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.231164 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.231142 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.232836 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.232812 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.233368 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.233347 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.239322 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.239301 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8w2h\" (UniqueName: \"kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h\") pod \"auth-test-model-41807c-kserve-78b8dc747-kpb8k\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.247359 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.247341 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:57:29.323688 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.323666 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:29.403562 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.403535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:29.451866 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.451837 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 04:57:29.454812 ip-10-0-142-139 kubenswrapper[2560]: W0707 04:57:29.454785 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bc6aad1_1e69_4f7a_819f_ebd1e481d22c.slice/crio-115f7354c9885f57a24cf3fbb193e99f5fad5a0d62d4b18bc854016df07723ca WatchSource:0}: Error finding container 115f7354c9885f57a24cf3fbb193e99f5fad5a0d62d4b18bc854016df07723ca: Status 404 returned error can't find the container with id 115f7354c9885f57a24cf3fbb193e99f5fad5a0d62d4b18bc854016df07723ca Jul 07 04:57:29.500592 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:29.500564 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:30.058451 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:30.058418 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" event={"ID":"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c","Type":"ContainerStarted","Data":"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536"} Jul 07 04:57:30.058451 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:30.058454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" event={"ID":"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c","Type":"ContainerStarted","Data":"115f7354c9885f57a24cf3fbb193e99f5fad5a0d62d4b18bc854016df07723ca"} Jul 07 04:57:30.076933 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:30.076886 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" podStartSLOduration=2.076872496 podStartE2EDuration="2.076872496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 04:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 04:57:30.07585253 +0000 UTC m=+2007.743270723" watchObservedRunningTime="2026-07-07 04:57:30.076872496 +0000 UTC m=+2007.744290687" Jul 07 04:57:37.828141 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:37.828102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:37.943648 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:37.943616 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:39.324588 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:39.324557 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:39.324971 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:39.324605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:39.336781 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:39.336752 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:40.113470 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:40.113438 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:57:40.118266 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:40.118243 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 04:57:55.323505 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:55.323471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:57:58.837604 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:57:58.832656 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:58:24.102789 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:24.102750 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:58:26.595430 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:26.595399 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:58:38.302847 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:38.302810 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:58:48.479134 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:48.479091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:58:48.529529 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:48.529500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:58:50.582401 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:50.582363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:58:54.996848 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:58:54.996813 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:59:03.304719 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:03.304696 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:59:03.312110 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:03.312090 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 04:59:12.002680 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:12.002638 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:59:27.708576 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:27.708532 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:59:42.002534 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:42.002497 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 04:59:56.048811 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:56.048740 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 04:59:56.070573 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:56.070544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:59:56.916574 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:56.916538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 04:59:59.102673 ip-10-0-142-139 kubenswrapper[2560]: I0707 04:59:59.102635 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:00:00.145489 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.145442 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz"] Jul 07 05:00:00.148790 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.148769 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7"] Jul 07 05:00:00.148933 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.148897 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:00.151429 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.151410 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-fjrxt\"" Jul 07 05:00:00.152463 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.152439 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:00.157567 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.157547 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz"] Jul 07 05:00:00.172526 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.172502 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7"] Jul 07 05:00:00.293387 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.293358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9dd\" (UniqueName: \"kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd\") pod \"maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7\" (UID: \"93763c4e-04d0-465a-a01b-e26de70efb01\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:00.293565 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.293479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgxt\" (UniqueName: \"kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt\") pod \"maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz\" (UID: \"36f73a42-0447-4c0f-9d3f-52c91ec37812\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:00.394459 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.394430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgxt\" (UniqueName: \"kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt\") pod \"maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz\" (UID: \"36f73a42-0447-4c0f-9d3f-52c91ec37812\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:00.394578 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.394473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9dd\" (UniqueName: \"kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd\") pod \"maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7\" (UID: \"93763c4e-04d0-465a-a01b-e26de70efb01\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:00.402905 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.402852 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9dd\" (UniqueName: \"kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd\") pod \"maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7\" (UID: \"93763c4e-04d0-465a-a01b-e26de70efb01\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:00.403019 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.402946 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgxt\" (UniqueName: \"kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt\") pod \"maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz\" (UID: \"36f73a42-0447-4c0f-9d3f-52c91ec37812\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:00.461562 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.461541 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:00.468389 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.468370 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:00.602539 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.602512 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz"] Jul 07 05:00:00.607505 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:00:00.607477 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f73a42_0447_4c0f_9d3f_52c91ec37812.slice/crio-7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413 WatchSource:0}: Error finding container 7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413: Status 404 returned error can't find the container with id 7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413 Jul 07 05:00:00.609433 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.609415 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 05:00:00.621769 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.621742 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7"] Jul 07 05:00:00.626173 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:00:00.626150 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93763c4e_04d0_465a_a01b_e26de70efb01.slice/crio-09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50 WatchSource:0}: Error finding container 09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50: Status 404 returned error can't find the container with id 09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50 Jul 07 05:00:00.787287 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.787209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" event={"ID":"36f73a42-0447-4c0f-9d3f-52c91ec37812","Type":"ContainerStarted","Data":"7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413"} Jul 07 05:00:00.788413 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:00.788387 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" event={"ID":"93763c4e-04d0-465a-a01b-e26de70efb01","Type":"ContainerStarted","Data":"09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50"} Jul 07 05:00:01.082991 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.082960 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:00:01.087565 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.087542 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.090278 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.090257 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-41807c\"/\"sub-test-model-41807c-kserve-self-signed-certs\"" Jul 07 05:00:01.096744 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.096722 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:00:01.129163 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.129133 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:00:01.200341 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.200653 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200352 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.200653 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.200653 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sg7\" (UniqueName: \"kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.200653 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.200653 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.200537 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301523 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sg7\" (UniqueName: \"kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301557 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301578 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.301703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.301641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.302090 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.302068 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.302213 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.302130 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.302213 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.302140 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.303781 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.303758 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.304045 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.304027 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.309897 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.309877 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sg7\" (UniqueName: \"kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7\") pod \"sub-test-model-41807c-kserve-7f9bd7849-dbqzt\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.398597 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.398547 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:01.526544 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.526519 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:00:01.528090 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:00:01.528062 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcf555c_c981_4d38_87ff_06d3fa0c8124.slice/crio-8c7fd529a496d3ea60073d3b47ace92d5d4a806ae25900ca4c7f4e5c46dfbc47 WatchSource:0}: Error finding container 8c7fd529a496d3ea60073d3b47ace92d5d4a806ae25900ca4c7f4e5c46dfbc47: Status 404 returned error can't find the container with id 8c7fd529a496d3ea60073d3b47ace92d5d4a806ae25900ca4c7f4e5c46dfbc47 Jul 07 05:00:01.795380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.795294 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" event={"ID":"6dcf555c-c981-4d38-87ff-06d3fa0c8124","Type":"ContainerStarted","Data":"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1"} Jul 07 05:00:01.795380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.795342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" event={"ID":"6dcf555c-c981-4d38-87ff-06d3fa0c8124","Type":"ContainerStarted","Data":"8c7fd529a496d3ea60073d3b47ace92d5d4a806ae25900ca4c7f4e5c46dfbc47"} Jul 07 05:00:01.815193 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:01.815142 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" podStartSLOduration=0.815128417 podStartE2EDuration="815.128417ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 05:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:00:01.812993395 +0000 UTC m=+2159.480411590" watchObservedRunningTime="2026-07-07 05:00:01.815128417 +0000 UTC m=+2159.482546604" Jul 07 05:00:04.813614 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:04.813579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" event={"ID":"36f73a42-0447-4c0f-9d3f-52c91ec37812","Type":"ContainerStarted","Data":"abc911798769baf8da2b88bcf20e523b8704102ffd24bb1d151ea75b030e165d"} Jul 07 05:00:04.815027 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:04.814991 2560 generic.go:356] "Generic (PLEG): container finished" podID="93763c4e-04d0-465a-a01b-e26de70efb01" containerID="b6ff45bb4a29130d6bfc99981f374571d04368488ef1cd0512261dd7c92c16e6" exitCode=0 Jul 07 05:00:04.815116 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:04.815054 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" event={"ID":"93763c4e-04d0-465a-a01b-e26de70efb01","Type":"ContainerDied","Data":"b6ff45bb4a29130d6bfc99981f374571d04368488ef1cd0512261dd7c92c16e6"} Jul 07 05:00:04.829608 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:04.829564 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" podStartSLOduration=1.513982671 podStartE2EDuration="4.829550764s" totalImagesPullingTime="3.315568093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 05:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:00:04.828829795 +0000 UTC m=+2162.496248004" watchObservedRunningTime="2026-07-07 05:00:04.829550764 +0000 UTC m=+2162.496968954" Jul 07 05:00:05.821250 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:05.821212 2560 generic.go:356] "Generic (PLEG): container finished" podID="36f73a42-0447-4c0f-9d3f-52c91ec37812" containerID="abc911798769baf8da2b88bcf20e523b8704102ffd24bb1d151ea75b030e165d" exitCode=0 Jul 07 05:00:05.821641 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:05.821297 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" event={"ID":"36f73a42-0447-4c0f-9d3f-52c91ec37812","Type":"ContainerDied","Data":"abc911798769baf8da2b88bcf20e523b8704102ffd24bb1d151ea75b030e165d"} Jul 07 05:00:05.948998 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:05.948972 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:06.041455 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.041424 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j9dd\" (UniqueName: \"kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd\") pod \"93763c4e-04d0-465a-a01b-e26de70efb01\" (UID: \"93763c4e-04d0-465a-a01b-e26de70efb01\") " Jul 07 05:00:06.043550 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.043524 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd" (OuterVolumeSpecName: "kube-api-access-6j9dd") pod "93763c4e-04d0-465a-a01b-e26de70efb01" (UID: "93763c4e-04d0-465a-a01b-e26de70efb01"). InnerVolumeSpecName "kube-api-access-6j9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:00:06.143041 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.142972 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9dd\" (UniqueName: \"kubernetes.io/projected/93763c4e-04d0-465a-a01b-e26de70efb01-kube-api-access-6j9dd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:06.832767 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.832734 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" Jul 07 05:00:06.833195 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.832763 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7" event={"ID":"93763c4e-04d0-465a-a01b-e26de70efb01","Type":"ContainerDied","Data":"09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50"} Jul 07 05:00:06.833195 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.832836 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c1a04f21b74aa066d59c2ed612b764f15731e754040f2a2b50c12cb855bf50" Jul 07 05:00:06.957733 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:06.957711 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:07.050414 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.050377 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjgxt\" (UniqueName: \"kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt\") pod \"36f73a42-0447-4c0f-9d3f-52c91ec37812\" (UID: \"36f73a42-0447-4c0f-9d3f-52c91ec37812\") " Jul 07 05:00:07.052319 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.052276 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt" (OuterVolumeSpecName: "kube-api-access-vjgxt") pod "36f73a42-0447-4c0f-9d3f-52c91ec37812" (UID: "36f73a42-0447-4c0f-9d3f-52c91ec37812"). InnerVolumeSpecName "kube-api-access-vjgxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:00:07.151168 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.151110 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjgxt\" (UniqueName: \"kubernetes.io/projected/36f73a42-0447-4c0f-9d3f-52c91ec37812-kube-api-access-vjgxt\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:07.839567 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.839534 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" event={"ID":"36f73a42-0447-4c0f-9d3f-52c91ec37812","Type":"ContainerDied","Data":"7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413"} Jul 07 05:00:07.839966 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.839580 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfe32761c35698720a78d99ddd76382eba9dd55652b2d6edc2606fcbc7d9413" Jul 07 05:00:07.839966 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:07.839543 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz" Jul 07 05:00:11.069848 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.069807 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" podUID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" containerName="main" containerID="cri-o://22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536" gracePeriod=45 Jul 07 05:00:11.314766 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.314743 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 05:00:11.399721 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.399661 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:11.399721 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.399704 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:11.411563 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.411543 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:11.484995 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.484971 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485115 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485027 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485115 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485059 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485115 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485090 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485121 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485196 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8w2h\" (UniqueName: \"kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h\") pod \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\" (UID: \"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c\") " Jul 07 05:00:11.485273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485249 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:00:11.485439 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485360 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:00:11.485503 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485447 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home" (OuterVolumeSpecName: "home") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:00:11.485541 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485506 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.485541 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.485523 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.487261 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.487235 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:00:11.487357 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.487264 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm" (OuterVolumeSpecName: "dshm") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:00:11.487357 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.487266 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h" (OuterVolumeSpecName: "kube-api-access-q8w2h") pod "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" (UID: "0bc6aad1-1e69-4f7a-819f-ebd1e481d22c"). InnerVolumeSpecName "kube-api-access-q8w2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:00:11.586604 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.586579 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.586604 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.586600 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.586777 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.586613 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.586777 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.586624 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8w2h\" (UniqueName: \"kubernetes.io/projected/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c-kube-api-access-q8w2h\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:00:11.861369 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.861340 2560 generic.go:356] "Generic (PLEG): container finished" podID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" containerID="22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536" exitCode=0 Jul 07 05:00:11.861612 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.861383 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" Jul 07 05:00:11.861612 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.861419 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" event={"ID":"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c","Type":"ContainerDied","Data":"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536"} Jul 07 05:00:11.861612 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.861464 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k" event={"ID":"0bc6aad1-1e69-4f7a-819f-ebd1e481d22c","Type":"ContainerDied","Data":"115f7354c9885f57a24cf3fbb193e99f5fad5a0d62d4b18bc854016df07723ca"} Jul 07 05:00:11.861612 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.861482 2560 scope.go:117] "RemoveContainer" containerID="22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536" Jul 07 05:00:11.871870 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.871856 2560 scope.go:117] "RemoveContainer" containerID="22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536" Jul 07 05:00:11.872145 ip-10-0-142-139 kubenswrapper[2560]: E0707 05:00:11.872128 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536\": container with ID starting with 22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536 not found: ID does not exist" containerID="22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536" Jul 07 05:00:11.872188 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.872154 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536"} err="failed to get container status \"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536\": rpc error: code = NotFound desc = could not find container \"22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536\": container with ID starting with 22bec0d580df26f3359ee031ca7797e756d23fe41dd0c8543360818f93624536 not found: ID does not exist" Jul 07 05:00:11.873418 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.873391 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:00:11.884386 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.884364 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 05:00:11.890632 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:11.890614 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/auth-test-model-41807c-kserve-78b8dc747-kpb8k"] Jul 07 05:00:13.235940 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:13.235908 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" path="/var/lib/kubelet/pods/0bc6aad1-1e69-4f7a-819f-ebd1e481d22c/volumes" Jul 07 05:00:13.389200 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:13.389166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:00:13.489711 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:13.489618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:00:14.397877 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:14.397845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:00:21.582383 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582352 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582661 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" containerName="main" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582671 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" containerName="main" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582698 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93763c4e-04d0-465a-a01b-e26de70efb01" containerName="cleanup" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582704 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="93763c4e-04d0-465a-a01b-e26de70efb01" containerName="cleanup" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582709 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36f73a42-0447-4c0f-9d3f-52c91ec37812" containerName="cleanup" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582714 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="36f73a42-0447-4c0f-9d3f-52c91ec37812" containerName="cleanup" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582768 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc6aad1-1e69-4f7a-819f-ebd1e481d22c" containerName="main" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582776 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="93763c4e-04d0-465a-a01b-e26de70efb01" containerName="cleanup" Jul 07 05:00:21.582849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.582783 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="36f73a42-0447-4c0f-9d3f-52c91ec37812" containerName="cleanup" Jul 07 05:00:21.587029 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.586987 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.589726 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.589708 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"kube-root-ca.crt\"" Jul 07 05:00:21.591059 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.591030 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"openshift-service-ca.crt\"" Jul 07 05:00:21.591293 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.591268 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"sub-test-model-5a77f6-kserve-self-signed-certs\"" Jul 07 05:00:21.591386 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.591359 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"default-dockercfg-2nv8p\"" Jul 07 05:00:21.597818 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.596974 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:00:21.661184 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661158 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.661275 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.661275 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26pb\" (UniqueName: \"kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.661349 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661298 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.661388 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661352 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.661422 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.661387 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.699634 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.699597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:00:21.762597 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762575 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.762694 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762614 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.762694 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762636 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26pb\" (UniqueName: \"kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.762694 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.762849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762702 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.762849 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.762727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.763061 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.763043 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.763122 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.763079 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.763158 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.763119 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.764941 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.764920 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.765089 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.765075 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.771099 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.771078 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26pb\" (UniqueName: \"kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb\") pod \"sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:21.901603 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:21.901531 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:22.033114 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:22.032964 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:00:22.035338 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:00:22.035303 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda41d8ea2_a29b_4b42_907b_037371c6ef1f.slice/crio-555ff21b05ddb927d2ca894403e30bb940537b6e41bb79aff4af552bc27e9565 WatchSource:0}: Error finding container 555ff21b05ddb927d2ca894403e30bb940537b6e41bb79aff4af552bc27e9565: Status 404 returned error can't find the container with id 555ff21b05ddb927d2ca894403e30bb940537b6e41bb79aff4af552bc27e9565 Jul 07 05:00:22.920856 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:22.920813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" event={"ID":"a41d8ea2-a29b-4b42-907b-037371c6ef1f","Type":"ContainerStarted","Data":"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929"} Jul 07 05:00:22.920856 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:22.920868 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" event={"ID":"a41d8ea2-a29b-4b42-907b-037371c6ef1f","Type":"ContainerStarted","Data":"555ff21b05ddb927d2ca894403e30bb940537b6e41bb79aff4af552bc27e9565"} Jul 07 05:00:22.938896 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:22.938848 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" podStartSLOduration=1.93883396 podStartE2EDuration="1.93883396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 05:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:00:22.93729131 +0000 UTC m=+2180.604709502" watchObservedRunningTime="2026-07-07 05:00:22.93883396 +0000 UTC m=+2180.606252150" Jul 07 05:00:27.998783 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:27.998747 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:00:31.902564 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:31.902530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:31.902962 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:31.902578 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:31.914963 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:31.914932 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:31.975536 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:31.975507 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:00:46.607069 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:00:46.607031 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:01:04.500581 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:04.500544 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:01:10.833608 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:10.833574 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:01:11.076786 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:11.076748 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:01:11.670897 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:11.670870 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:01:11.700590 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:11.700565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:01:14.276740 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:14.276708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:01:22.799781 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:22.799746 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:01:25.853756 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:25.853664 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" podUID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" containerName="main" containerID="cri-o://b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1" gracePeriod=45 Jul 07 05:01:26.103912 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.103846 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:01:26.206783 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206735 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4sg7\" (UniqueName: \"kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.206977 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206837 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.206977 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206864 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.206977 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206902 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.206977 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206927 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.206977 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.206955 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home\") pod \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\" (UID: \"6dcf555c-c981-4d38-87ff-06d3fa0c8124\") " Jul 07 05:01:26.207282 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.207181 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache" (OuterVolumeSpecName: "model-cache") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:26.207282 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.207228 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:26.207282 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.207246 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home" (OuterVolumeSpecName: "home") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:26.208932 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.208905 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7" (OuterVolumeSpecName: "kube-api-access-h4sg7") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "kube-api-access-h4sg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:01:26.209060 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.208937 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:01:26.209060 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.208936 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm" (OuterVolumeSpecName: "dshm") pod "6dcf555c-c981-4d38-87ff-06d3fa0c8124" (UID: "6dcf555c-c981-4d38-87ff-06d3fa0c8124"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:26.237120 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.237091 2560 generic.go:356] "Generic (PLEG): container finished" podID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" containerID="b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1" exitCode=0 Jul 07 05:01:26.237259 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.237136 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" Jul 07 05:01:26.237259 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.237177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" event={"ID":"6dcf555c-c981-4d38-87ff-06d3fa0c8124","Type":"ContainerDied","Data":"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1"} Jul 07 05:01:26.237259 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.237222 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt" event={"ID":"6dcf555c-c981-4d38-87ff-06d3fa0c8124","Type":"ContainerDied","Data":"8c7fd529a496d3ea60073d3b47ace92d5d4a806ae25900ca4c7f4e5c46dfbc47"} Jul 07 05:01:26.237368 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.237244 2560 scope.go:117] "RemoveContainer" containerID="b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1" Jul 07 05:01:26.248390 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.248371 2560 scope.go:117] "RemoveContainer" containerID="b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1" Jul 07 05:01:26.248638 ip-10-0-142-139 kubenswrapper[2560]: E0707 05:01:26.248622 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1\": container with ID starting with b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1 not found: ID does not exist" containerID="b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1" Jul 07 05:01:26.248690 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.248647 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1"} err="failed to get container status \"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1\": rpc error: code = NotFound desc = could not find container \"b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1\": container with ID starting with b1a1dd4fe605df9672c8411dc51aa01ef1af966d3a6ddf5c24f18da70384d5f1 not found: ID does not exist" Jul 07 05:01:26.261523 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.261502 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:01:26.263431 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.263410 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-41807c/sub-test-model-41807c-kserve-7f9bd7849-dbqzt"] Jul 07 05:01:26.307565 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307535 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.307565 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307562 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.307565 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307570 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.307764 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307577 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.307764 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307585 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf555c-c981-4d38-87ff-06d3fa0c8124-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.307764 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.307593 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sg7\" (UniqueName: \"kubernetes.io/projected/6dcf555c-c981-4d38-87ff-06d3fa0c8124-kube-api-access-h4sg7\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:26.690807 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.690763 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" podUID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" containerName="main" containerID="cri-o://95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929" gracePeriod=45 Jul 07 05:01:26.936655 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:26.936630 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:01:27.014039 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.013950 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014039 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.013978 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014039 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014031 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k26pb\" (UniqueName: \"kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014289 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014086 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014289 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014124 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014289 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014148 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache\") pod \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\" (UID: \"a41d8ea2-a29b-4b42-907b-037371c6ef1f\") " Jul 07 05:01:27.014289 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014219 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home" (OuterVolumeSpecName: "home") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:27.014481 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014386 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.014481 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014396 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:27.014481 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.014422 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:27.016186 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.016159 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm" (OuterVolumeSpecName: "dshm") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:01:27.016309 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.016216 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:01:27.016309 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.016277 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb" (OuterVolumeSpecName: "kube-api-access-k26pb") pod "a41d8ea2-a29b-4b42-907b-037371c6ef1f" (UID: "a41d8ea2-a29b-4b42-907b-037371c6ef1f"). InnerVolumeSpecName "kube-api-access-k26pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:01:27.115087 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.115052 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.115087 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.115083 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26pb\" (UniqueName: \"kubernetes.io/projected/a41d8ea2-a29b-4b42-907b-037371c6ef1f-kube-api-access-k26pb\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.115273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.115097 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.115273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.115110 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41d8ea2-a29b-4b42-907b-037371c6ef1f-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.115273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.115123 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41d8ea2-a29b-4b42-907b-037371c6ef1f-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:01:27.235861 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.235834 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" path="/var/lib/kubelet/pods/6dcf555c-c981-4d38-87ff-06d3fa0c8124/volumes" Jul 07 05:01:27.244335 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.244264 2560 generic.go:356] "Generic (PLEG): container finished" podID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" containerID="95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929" exitCode=0 Jul 07 05:01:27.244335 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.244308 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" event={"ID":"a41d8ea2-a29b-4b42-907b-037371c6ef1f","Type":"ContainerDied","Data":"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929"} Jul 07 05:01:27.244477 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.244360 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" event={"ID":"a41d8ea2-a29b-4b42-907b-037371c6ef1f","Type":"ContainerDied","Data":"555ff21b05ddb927d2ca894403e30bb940537b6e41bb79aff4af552bc27e9565"} Jul 07 05:01:27.244477 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.244378 2560 scope.go:117] "RemoveContainer" containerID="95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929" Jul 07 05:01:27.244477 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.244318 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp" Jul 07 05:01:27.252581 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.252564 2560 scope.go:117] "RemoveContainer" containerID="95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929" Jul 07 05:01:27.252824 ip-10-0-142-139 kubenswrapper[2560]: E0707 05:01:27.252806 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929\": container with ID starting with 95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929 not found: ID does not exist" containerID="95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929" Jul 07 05:01:27.252877 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.252833 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929"} err="failed to get container status \"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929\": rpc error: code = NotFound desc = could not find container \"95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929\": container with ID starting with 95f919a83d2f4bfe15744e7d61067d62ea6b772b0896a6079a55958ae4d81929 not found: ID does not exist" Jul 07 05:01:27.262031 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.261990 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:01:27.268884 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:27.268830 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/sub-test-model-5a77f6-kserve-7d567fd69f-xfjcp"] Jul 07 05:01:29.236102 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:29.236075 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" path="/var/lib/kubelet/pods/a41d8ea2-a29b-4b42-907b-037371c6ef1f/volumes" Jul 07 05:01:57.489936 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.489896 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490226 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" containerName="main" Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490237 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" containerName="main" Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490256 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" containerName="main" Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490261 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" containerName="main" Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490321 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcf555c-c981-4d38-87ff-06d3fa0c8124" containerName="main" Jul 07 05:01:57.490435 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.490332 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41d8ea2-a29b-4b42-907b-037371c6ef1f" containerName="main" Jul 07 05:01:57.493245 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.493227 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.497076 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.497054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"default-dockercfg-2nv8p\"" Jul 07 05:01:57.497194 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.497081 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"openshift-service-ca.crt\"" Jul 07 05:01:57.497194 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.497054 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"kube-root-ca.crt\"" Jul 07 05:01:57.497194 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.497056 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a77f6\"/\"rate-test-model-5a77f6-kserve-self-signed-certs\"" Jul 07 05:01:57.501412 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.501390 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:01:57.543999 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.543970 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:01:57.571332 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571300 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.571453 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.571453 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571445 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.571532 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.571532 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571496 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.571532 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.571513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzkd\" (UniqueName: \"kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672689 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672689 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672970 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672714 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672970 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzkd\" (UniqueName: \"kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672970 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672783 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.672970 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.672802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.673208 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.673188 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.673290 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.673224 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.673290 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.673283 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.675094 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.675067 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.675281 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.675263 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.682876 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.682853 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzkd\" (UniqueName: \"kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd\") pod \"rate-test-model-5a77f6-kserve-86b8c94479-79sg9\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.803135 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.803033 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:01:57.934491 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:57.934463 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:01:57.937729 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:01:57.937098 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac24ea61_d7c3_4a95_b3a3_bf991913f5f6.slice/crio-e6851d023f7b2a0f9dd3759512a18ce4f8326e9aab4ee7ecbb51bfad2e34b0ac WatchSource:0}: Error finding container e6851d023f7b2a0f9dd3759512a18ce4f8326e9aab4ee7ecbb51bfad2e34b0ac: Status 404 returned error can't find the container with id e6851d023f7b2a0f9dd3759512a18ce4f8326e9aab4ee7ecbb51bfad2e34b0ac Jul 07 05:01:58.397753 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:58.397712 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" event={"ID":"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6","Type":"ContainerStarted","Data":"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de"} Jul 07 05:01:58.397955 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:58.397762 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" event={"ID":"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6","Type":"ContainerStarted","Data":"e6851d023f7b2a0f9dd3759512a18ce4f8326e9aab4ee7ecbb51bfad2e34b0ac"} Jul 07 05:01:58.417564 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:01:58.417517 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" podStartSLOduration=1.417502174 podStartE2EDuration="1.417502174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 05:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:01:58.416051955 +0000 UTC m=+2276.083470143" watchObservedRunningTime="2026-07-07 05:01:58.417502174 +0000 UTC m=+2276.084920366" Jul 07 05:02:01.798283 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:01.798244 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:02:07.803132 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:07.803091 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:07.803550 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:07.803159 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:07.815381 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:07.815359 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:08.457364 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:08.457336 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:15.198986 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:15.198952 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:02:16.106273 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:16.106234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:18.303942 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:18.303902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:18.442736 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:18.442700 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:29.795751 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:29.795709 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:02:30.307354 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:30.307319 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:31.806405 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:31.806372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:36.020391 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:36.020357 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:36.794918 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:36.794883 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:02:36.816852 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:36.816823 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:37.885714 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:37.885682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:38.337662 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:38.337624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:38.863509 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:38.863473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:38.967259 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:38.967221 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:39.108505 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:39.108456 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7"] Jul 07 05:02:39.110530 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:39.110507 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-41807c-29723340-c8gl7"] Jul 07 05:02:39.111108 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:39.111088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:39.235986 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:39.235907 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93763c4e-04d0-465a-a01b-e26de70efb01" path="/var/lib/kubelet/pods/93763c4e-04d0-465a-a01b-e26de70efb01/volumes" Jul 07 05:02:43.384738 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:43.384707 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:02:43.598651 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:43.598618 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:02:51.816105 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:51.816046 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" podUID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" containerName="main" containerID="cri-o://1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de" gracePeriod=45 Jul 07 05:02:52.062699 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.062673 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:52.145716 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145621 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.145716 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145669 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.145716 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145691 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.145978 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145745 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rzkd\" (UniqueName: \"kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.145978 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145766 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.145978 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145790 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm\") pod \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\" (UID: \"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6\") " Jul 07 05:02:52.146181 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.146037 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:02:52.146181 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.145976 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:02:52.146181 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.146066 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home" (OuterVolumeSpecName: "home") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:02:52.147862 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.147825 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm" (OuterVolumeSpecName: "dshm") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 05:02:52.147967 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.147859 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 05:02:52.147967 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.147902 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd" (OuterVolumeSpecName: "kube-api-access-7rzkd") pod "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" (UID: "ac24ea61-d7c3-4a95-b3a3-bf991913f5f6"). InnerVolumeSpecName "kube-api-access-7rzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 05:02:52.247059 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.246997 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rzkd\" (UniqueName: \"kubernetes.io/projected/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-kube-api-access-7rzkd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.247059 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.247049 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.247059 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.247059 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.247059 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.247066 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.247336 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.247075 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.247336 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.247083 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 07 05:02:52.663645 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.663602 2560 generic.go:356] "Generic (PLEG): container finished" podID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" containerID="1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de" exitCode=0 Jul 07 05:02:52.663645 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.663649 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" Jul 07 05:02:52.663897 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.663646 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" event={"ID":"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6","Type":"ContainerDied","Data":"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de"} Jul 07 05:02:52.663897 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.663771 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9" event={"ID":"ac24ea61-d7c3-4a95-b3a3-bf991913f5f6","Type":"ContainerDied","Data":"e6851d023f7b2a0f9dd3759512a18ce4f8326e9aab4ee7ecbb51bfad2e34b0ac"} Jul 07 05:02:52.663897 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.663798 2560 scope.go:117] "RemoveContainer" containerID="1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de" Jul 07 05:02:52.673093 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.673070 2560 scope.go:117] "RemoveContainer" containerID="1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de" Jul 07 05:02:52.673382 ip-10-0-142-139 kubenswrapper[2560]: E0707 05:02:52.673360 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de\": container with ID starting with 1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de not found: ID does not exist" containerID="1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de" Jul 07 05:02:52.673427 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.673392 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de"} err="failed to get container status \"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de\": rpc error: code = NotFound desc = could not find container \"1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de\": container with ID starting with 1a4c9db23f9e64409a4b5925e6d60990197d2e59d250c28ae0ceac79eecf69de not found: ID does not exist" Jul 07 05:02:52.685703 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.685669 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:02:52.691950 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:52.691919 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a77f6/rate-test-model-5a77f6-kserve-86b8c94479-79sg9"] Jul 07 05:02:53.236554 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:02:53.236526 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" path="/var/lib/kubelet/pods/ac24ea61-d7c3-4a95-b3a3-bf991913f5f6/volumes" Jul 07 05:03:20.221386 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:20.221350 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:20.493116 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:20.493025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:20.729518 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:20.729484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:20.900701 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:20.900662 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:21.205814 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:21.205730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:21.211497 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:21.211460 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz"] Jul 07 05:03:21.213335 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:21.213317 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5a77f6-29723340-96lwz"] Jul 07 05:03:21.236021 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:21.235977 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36f73a42-0447-4c0f-9d3f-52c91ec37812" path="/var/lib/kubelet/pods/36f73a42-0447-4c0f-9d3f-52c91ec37812/volumes" Jul 07 05:03:21.307200 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:21.307164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:23.400164 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:23.400128 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7mv4m"] Jul 07 05:03:27.353256 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:27.353225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:27.452719 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:27.452686 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:27.619094 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:27.618972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:27.861463 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:27.861425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:28.283700 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:28.283660 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95d4cf77-jvj9g_5643f5ec-ddea-4c27-8aec-3c7677752883/authorino/0.log" Jul 07 05:03:33.646131 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:33.646098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-25zzt_bef55b86-d4c8-4639-b219-fc153e8f18df/server/0.log" Jul 07 05:03:33.763390 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:33.763352 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-crmjc_86cb3fd7-08fa-4b29-a5fd-2c5467036635/manager/0.log" Jul 07 05:03:33.890555 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:33.890521 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-fd69k_22a8a364-c538-47c5-80de-86bc163e6ea1/copy-manifests/0.log" Jul 07 05:03:33.898036 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:33.897947 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-fd69k_22a8a364-c538-47c5-80de-86bc163e6ea1/manager/0.log" Jul 07 05:03:35.044469 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.044435 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/util/0.log" Jul 07 05:03:35.051088 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.051062 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/pull/0.log" Jul 07 05:03:35.057971 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.057951 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/extract/0.log" Jul 07 05:03:35.167657 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.167626 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/util/0.log" Jul 07 05:03:35.174101 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.174076 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/pull/0.log" Jul 07 05:03:35.180414 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.180393 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/extract/0.log" Jul 07 05:03:35.292155 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.292126 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/util/0.log" Jul 07 05:03:35.298676 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.298603 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/pull/0.log" Jul 07 05:03:35.305212 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.305165 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/extract/0.log" Jul 07 05:03:35.419047 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.419017 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/pull/0.log" Jul 07 05:03:35.425900 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.425878 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/extract/0.log" Jul 07 05:03:35.432813 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.432784 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/util/0.log" Jul 07 05:03:35.547753 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.547718 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95d4cf77-jvj9g_5643f5ec-ddea-4c27-8aec-3c7677752883/authorino/0.log" Jul 07 05:03:35.891755 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:35.891729 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9ngqr_588265d9-beb7-4dfb-8105-b742f2ccafab/kuadrant-console-plugin/0.log" Jul 07 05:03:36.265655 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:36.265574 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7mv4m_1a6f4e9d-88b6-4b74-8641-3563c7609168/limitador/0.log" Jul 07 05:03:37.093342 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:37.093314 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-lgpsv_505c6df7-6d97-41cb-a430-2d93f65e9f6f/kube-auth-proxy/0.log" Jul 07 05:03:42.733442 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.733400 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jh4c7/must-gather-r5svn"] Jul 07 05:03:42.734034 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.733950 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" containerName="main" Jul 07 05:03:42.734034 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.733970 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" containerName="main" Jul 07 05:03:42.734174 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.734073 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac24ea61-d7c3-4a95-b3a3-bf991913f5f6" containerName="main" Jul 07 05:03:42.737094 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.737068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.737289 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.737168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:42.739922 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.739901 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh4c7\"/\"openshift-service-ca.crt\"" Jul 07 05:03:42.740044 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.739965 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jh4c7\"/\"kube-root-ca.crt\"" Jul 07 05:03:42.740832 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.740815 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jh4c7\"/\"default-dockercfg-hblzf\"" Jul 07 05:03:42.745317 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.745297 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/must-gather-r5svn"] Jul 07 05:03:42.780165 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.780125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/2b6a38b1-613c-411d-b79f-1097e6e01d6c-kube-api-access-2bgkr\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.780165 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.780170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b6a38b1-613c-411d-b79f-1097e6e01d6c-must-gather-output\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.798840 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.798794 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" limit=1024 Jul 07 05:03:42.881215 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.881174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/2b6a38b1-613c-411d-b79f-1097e6e01d6c-kube-api-access-2bgkr\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.881215 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.881215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b6a38b1-613c-411d-b79f-1097e6e01d6c-must-gather-output\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.881561 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.881506 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b6a38b1-613c-411d-b79f-1097e6e01d6c-must-gather-output\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:42.890170 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:42.890145 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/2b6a38b1-613c-411d-b79f-1097e6e01d6c-kube-api-access-2bgkr\") pod \"must-gather-r5svn\" (UID: \"2b6a38b1-613c-411d-b79f-1097e6e01d6c\") " pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:43.047452 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:43.047354 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/must-gather-r5svn" Jul 07 05:03:43.193654 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:43.193629 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/must-gather-r5svn"] Jul 07 05:03:43.195496 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:03:43.195471 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b6a38b1_613c_411d_b79f_1097e6e01d6c.slice/crio-c307b50ea748cb773188864f83d8a5b5704f3d29f1ab03162ecbc295afe88d69 WatchSource:0}: Error finding container c307b50ea748cb773188864f83d8a5b5704f3d29f1ab03162ecbc295afe88d69: Status 404 returned error can't find the container with id c307b50ea748cb773188864f83d8a5b5704f3d29f1ab03162ecbc295afe88d69 Jul 07 05:03:43.924051 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:43.923993 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/must-gather-r5svn" event={"ID":"2b6a38b1-613c-411d-b79f-1097e6e01d6c","Type":"ContainerStarted","Data":"c307b50ea748cb773188864f83d8a5b5704f3d29f1ab03162ecbc295afe88d69"} Jul 07 05:03:44.941981 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:44.941852 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/must-gather-r5svn" event={"ID":"2b6a38b1-613c-411d-b79f-1097e6e01d6c","Type":"ContainerStarted","Data":"a55b46c17b9d172f4bf3333c9b86386187acee875778fe9496da3c374e02baab"} Jul 07 05:03:44.941981 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:44.941900 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/must-gather-r5svn" event={"ID":"2b6a38b1-613c-411d-b79f-1097e6e01d6c","Type":"ContainerStarted","Data":"e0cac71464c0d666760490cb0125ac08ea947c2d6fb4c2399ebdb0bb605441a7"} Jul 07 05:03:44.960505 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:44.960452 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jh4c7/must-gather-r5svn" podStartSLOduration=1.5787677100000002 podStartE2EDuration="2.960435004s" totalImagesPullingTime="1.381667294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 05:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:03:44.958404682 +0000 UTC m=+2382.625822901" watchObservedRunningTime="2026-07-07 05:03:44.960435004 +0000 UTC m=+2382.627853194" Jul 07 05:03:46.212315 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:46.212283 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tx8w2_5ded6da4-58bc-4b73-b3fc-d47d9114e4f2/global-pull-secret-syncer/0.log" Jul 07 05:03:46.262815 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:46.262779 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gxjfc_534725e6-d911-4e9d-bee9-a90ceb824a86/konnectivity-agent/0.log" Jul 07 05:03:46.377457 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:46.377426 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-139.ec2.internal_7ebfeb482643ae23298dcd20cee13bf5/haproxy/0.log" Jul 07 05:03:50.989532 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:50.989505 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/extract/0.log" Jul 07 05:03:51.016155 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.016130 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/util/0.log" Jul 07 05:03:51.038635 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.038593 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zldzw_e954f984-8d91-41d3-89e4-0dde5003a4f8/pull/0.log" Jul 07 05:03:51.078581 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.078523 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/extract/0.log" Jul 07 05:03:51.101901 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.101861 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/util/0.log" Jul 07 05:03:51.126923 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.126897 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lrkxx_0cffe46c-9a71-4a8c-9c47-d02452a95b7c/pull/0.log" Jul 07 05:03:51.163407 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.163361 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/extract/0.log" Jul 07 05:03:51.186813 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.186788 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/util/0.log" Jul 07 05:03:51.208571 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.208516 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n5j7x_6fba0229-5715-4a47-8ff7-4e921363c525/pull/0.log" Jul 07 05:03:51.239473 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.239432 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/extract/0.log" Jul 07 05:03:51.260934 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.260819 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/util/0.log" Jul 07 05:03:51.283124 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.282994 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef12q4g8_e2ff68d5-d38e-478f-8836-3c85754308a8/pull/0.log" Jul 07 05:03:51.696759 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.696723 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95d4cf77-jvj9g_5643f5ec-ddea-4c27-8aec-3c7677752883/authorino/0.log" Jul 07 05:03:51.801100 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.801017 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9ngqr_588265d9-beb7-4dfb-8105-b742f2ccafab/kuadrant-console-plugin/0.log" Jul 07 05:03:51.977706 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:51.977549 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7mv4m_1a6f4e9d-88b6-4b74-8641-3563c7609168/limitador/0.log" Jul 07 05:03:53.740132 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.740047 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/alertmanager/0.log" Jul 07 05:03:53.768835 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.768810 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/config-reloader/0.log" Jul 07 05:03:53.790373 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.790337 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/kube-rbac-proxy-web/0.log" Jul 07 05:03:53.814343 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.814283 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/kube-rbac-proxy/0.log" Jul 07 05:03:53.837609 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.837576 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/kube-rbac-proxy-metric/0.log" Jul 07 05:03:53.862685 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.862656 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/prom-label-proxy/0.log" Jul 07 05:03:53.887629 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:53.887596 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e861919e-86ed-4b64-a450-b0500f152495/init-config-reloader/0.log" Jul 07 05:03:54.067326 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.067288 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-8b26m_b9181f5b-4c52-49ca-bb5d-a283bdd23769/monitoring-plugin/0.log" Jul 07 05:03:54.169490 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.169458 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvpvl_3cad0bc2-8287-4f29-ae47-6a53f4ed5b82/node-exporter/0.log" Jul 07 05:03:54.194983 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.194940 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvpvl_3cad0bc2-8287-4f29-ae47-6a53f4ed5b82/kube-rbac-proxy/0.log" Jul 07 05:03:54.221961 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.221931 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvpvl_3cad0bc2-8287-4f29-ae47-6a53f4ed5b82/init-textfile/0.log" Jul 07 05:03:54.630802 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.630776 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65bd4dbd5c-5wd82_5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd/telemeter-client/0.log" Jul 07 05:03:54.656347 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.656315 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65bd4dbd5c-5wd82_5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd/reload/0.log" Jul 07 05:03:54.676406 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.676377 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65bd4dbd5c-5wd82_5a0f08c6-b2de-4e25-9e3d-1f6176ef4ffd/kube-rbac-proxy/0.log" Jul 07 05:03:54.709206 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.709177 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/thanos-query/0.log" Jul 07 05:03:54.730566 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.730487 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/kube-rbac-proxy-web/0.log" Jul 07 05:03:54.756431 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.756396 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/kube-rbac-proxy/0.log" Jul 07 05:03:54.776647 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.776618 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/prom-label-proxy/0.log" Jul 07 05:03:54.800268 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.800222 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/kube-rbac-proxy-rules/0.log" Jul 07 05:03:54.822785 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:54.822752 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-697c7f855-rc92g_c1db23cf-71b6-4293-8837-21e139df2964/kube-rbac-proxy-metrics/0.log" Jul 07 05:03:56.519632 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.519594 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q"] Jul 07 05:03:56.525057 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.525031 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.533487 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.533451 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q"] Jul 07 05:03:56.613890 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.613846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-lib-modules\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.614224 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.614179 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5gc\" (UniqueName: \"kubernetes.io/projected/6e35b8c1-52aa-4a0b-9218-a206f64bb406-kube-api-access-hk5gc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.614428 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.614411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-sys\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.614587 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.614568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-proc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.614731 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.614715 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-podres\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716191 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-sys\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716209 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-proc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-podres\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716292 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-sys\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-lib-modules\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716349 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-proc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716380 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5gc\" (UniqueName: \"kubernetes.io/projected/6e35b8c1-52aa-4a0b-9218-a206f64bb406-kube-api-access-hk5gc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716685 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716417 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-lib-modules\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.716685 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.716417 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e35b8c1-52aa-4a0b-9218-a206f64bb406-podres\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.724927 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.724899 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5gc\" (UniqueName: \"kubernetes.io/projected/6e35b8c1-52aa-4a0b-9218-a206f64bb406-kube-api-access-hk5gc\") pod \"perf-node-gather-daemonset-bvg6q\" (UID: \"6e35b8c1-52aa-4a0b-9218-a206f64bb406\") " pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:56.838320 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:56.838284 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:57.019562 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:57.019524 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q"] Jul 07 05:03:57.023593 ip-10-0-142-139 kubenswrapper[2560]: W0707 05:03:57.022028 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e35b8c1_52aa_4a0b_9218_a206f64bb406.slice/crio-8f9e87bca7539a58dbdab5a7820ca35d3b4177bd288883e803d0486dcfdfb191 WatchSource:0}: Error finding container 8f9e87bca7539a58dbdab5a7820ca35d3b4177bd288883e803d0486dcfdfb191: Status 404 returned error can't find the container with id 8f9e87bca7539a58dbdab5a7820ca35d3b4177bd288883e803d0486dcfdfb191 Jul 07 05:03:57.033660 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:57.033630 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" event={"ID":"6e35b8c1-52aa-4a0b-9218-a206f64bb406","Type":"ContainerStarted","Data":"8f9e87bca7539a58dbdab5a7820ca35d3b4177bd288883e803d0486dcfdfb191"} Jul 07 05:03:58.042218 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.042180 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" event={"ID":"6e35b8c1-52aa-4a0b-9218-a206f64bb406","Type":"ContainerStarted","Data":"0e80bd3fa3d145b4833ce4285bc89c3637373fa980614a5ee6309c4bb8470471"} Jul 07 05:03:58.042636 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.042299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:03:58.074505 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.074442 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" podStartSLOduration=2.074422781 podStartE2EDuration="2.074422781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 05:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 05:03:58.0743846 +0000 UTC m=+2395.741802792" watchObservedRunningTime="2026-07-07 05:03:58.074422781 +0000 UTC m=+2395.741840974" Jul 07 05:03:58.719171 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.719141 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dqjvx_bef7a010-494d-478f-8ece-c4cf1debc5df/dns/0.log" Jul 07 05:03:58.760153 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.760122 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dqjvx_bef7a010-494d-478f-8ece-c4cf1debc5df/kube-rbac-proxy/0.log" Jul 07 05:03:58.852020 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:58.851975 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bnslv_cc1afd5c-8a3a-47a6-bdf1-5838616f1b6f/dns-node-resolver/0.log" Jul 07 05:03:59.441432 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:59.441398 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6df6899b44-mzhn4_0855b731-2cf2-43d4-8f02-90ba61c9c7f7/registry/0.log" Jul 07 05:03:59.505811 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:03:59.505781 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ttvr_ababb864-9ac2-497c-b428-cff38e27581a/node-ca/0.log" Jul 07 05:04:00.720813 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:00.720777 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-lgpsv_505c6df7-6d97-41cb-a430-2d93f65e9f6f/kube-auth-proxy/0.log" Jul 07 05:04:01.564489 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:01.564454 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lzpwn_afa7c157-2a1c-4421-8730-eae14f2fc659/serve-healthcheck-canary/0.log" Jul 07 05:04:02.060471 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:02.060383 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6hbj_d0683736-82a6-4cc2-badf-9f3228d7193d/kube-rbac-proxy/0.log" Jul 07 05:04:02.080635 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:02.080605 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6hbj_d0683736-82a6-4cc2-badf-9f3228d7193d/exporter/0.log" Jul 07 05:04:02.101641 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:02.101612 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k6hbj_d0683736-82a6-4cc2-badf-9f3228d7193d/extractor/0.log" Jul 07 05:04:03.348687 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:03.348656 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 05:04:03.356521 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:03.356495 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vdn94_8f94c1df-25d3-4528-9f2b-d7b3d3dce402/ovn-acl-logging/0.log" Jul 07 05:04:04.059694 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:04.059662 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jh4c7/perf-node-gather-daemonset-bvg6q" Jul 07 05:04:04.874812 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:04.874778 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-25zzt_bef55b86-d4c8-4639-b219-fc153e8f18df/server/0.log" Jul 07 05:04:04.960705 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:04.960666 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-crmjc_86cb3fd7-08fa-4b29-a5fd-2c5467036635/manager/0.log" Jul 07 05:04:04.991112 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:04.991080 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-fd69k_22a8a364-c538-47c5-80de-86bc163e6ea1/manager/0.log" Jul 07 05:04:05.013998 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:05.013967 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787b7c66b5-fd69k_22a8a364-c538-47c5-80de-86bc163e6ea1/copy-manifests/0.log" Jul 07 05:04:06.736533 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:06.736433 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-687876b94d-zp2ql_23f6f193-32eb-499b-8697-55120fb3f90d/manager/0.log" Jul 07 05:04:06.761132 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:06.761096 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-56rgt_cd42f678-7454-4bd1-a0ab-460094410392/openshift-lws-operator/0.log" Jul 07 05:04:07.187146 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:07.187111 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-lgpsv_505c6df7-6d97-41cb-a430-2d93f65e9f6f/kube-auth-proxy/0.log" Jul 07 05:04:08.731178 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:08.731090 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7db6fd6878-lgpsv_505c6df7-6d97-41cb-a430-2d93f65e9f6f/kube-auth-proxy/0.log" Jul 07 05:04:13.084981 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.084954 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/kube-multus-additional-cni-plugins/0.log" Jul 07 05:04:13.104979 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.104950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/egress-router-binary-copy/0.log" Jul 07 05:04:13.123829 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.123802 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/cni-plugins/0.log" Jul 07 05:04:13.143818 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.143788 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/bond-cni-plugin/0.log" Jul 07 05:04:13.163676 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.163647 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/routeoverride-cni/0.log" Jul 07 05:04:13.183509 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.183480 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/whereabouts-cni-bincopy/0.log" Jul 07 05:04:13.203659 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.203631 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7xfmc_b2a084a3-547a-4578-bd94-675420fca735/whereabouts-cni/0.log" Jul 07 05:04:13.608720 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.608691 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v7d99_c406620d-9bcd-476c-90b4-ec9e51b25c45/kube-multus/0.log" Jul 07 05:04:13.670402 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.670369 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdqmk_72012cd3-d8fa-46a2-90b8-32e2e92c4b6a/network-metrics-daemon/0.log" Jul 07 05:04:13.689482 ip-10-0-142-139 kubenswrapper[2560]: I0707 05:04:13.689451 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdqmk_72012cd3-d8fa-46a2-90b8-32e2e92c4b6a/kube-rbac-proxy/0.log"